Unexpected Risks Found in Editing Binary Code to Fix Bugs in Legacy Codebase.
Genetic engineering has exactly the same problems that modifying binary-only code. After some very complex observation of the system you may spot the point that is responsible for a certain problem, but you will never be sure (as in provably sure) that the "fix" you put in place:
1. is going to stop all the occurrences of the behavior you want to fix;
2. will only affect the behavior you want to fix;
3. will have no effect on other systems that you did not modify.
(The article is mostly about 1 and a bit about 2 and 3.)
Modifying legacy code is hard enough when the uncommented source code is around. Changing directly the binary is both a great engineering feat and something to be scared of.
Now imagine the code arose originally by random chance, happened to be self replicating and evolved over billions of generations to include mechanisms that both allow it to viable after small changes and allow those small changes to have large effects (due to selective pressure since diversity from mutations is advantageous to a point, especially in single celled organisms).
As an example, Some bacterial genomes are circular, code in both directions and have overlapping coding regions, and occasionally trade genetic material between species because why wouldn't they?
I've worked applying machine learning and statistics to genetics and I generally think it's pretty hilarious that many self declared rationalists think we need to worry about AI safety but declare scepticism of genetic engineering anti-scientific.
This comment and the parent comment are some of the most insightful things I've read on HN, thanks! Haven't considered genetic engineering's risk from a computational perspective, but this is absolutely spot on!
Unlike other experiments, genetic engineering can be incredibly disastrous to humanity. There's very real risks of destroying our food supply, the environment, and our health. And so far, we're taking these risks in the name of GMO crops, which are either resistant to toxic chemicals like glyphosate (Round-Up ready soybeans), or worse, plants that produce toxic chemicals to ward off pests (bt corn).
Genetic engineering and super-AI will eventually overlap. We'll use the new AI to be more capable in genetic engineering. And the new AI may use genetic engineering to empower itself in a more capable physical form than the hardware we know today.
> many self declared rationalists think we need to worry about AI safety but declare scepticism of genetic engineering anti-scientific.
It is because we've already done genetic engineering. Certainly there are tractability problems with the scope of a change and predicting it's full success, but the fact that changes can be made seems undeniable, even if all we do is stick to direct engineering and exclude the engineering we've been doing for thousands of years.
Protein folding is a computationally hard problem, so fully predicting a change's effect will remain imprecise, unless we find some new shortcuts, which is entirely possible.
My point is there are all sorts of hidden complexities and 2nd order interaction between things we don't yet understand.
I'd argue that this goes beyond computational tractability and is still an issue of statistical power. There are millions of common variants in the human genome and we've only sequenced thousands of people so it's not exactly controversial to say genetics is poorly understood.
Which as the AI-safety people say, isn't to say we should be researching it just that we should be devoting more energy to doing it safely as the potential consequences are so high even if the chances of things going wrong is low.
> Which as the AI-safety people say, isn't to say we should be researching it just that we should be devoting more energy to doing it safely as the potential consequences are so high even if the chances of things going wrong is low.
Are the potential consequences really so high? It's not like we're going to be genetically engineering all of humanity in the same way at once, whereas AI safety effects all of humanity at once.
I'd argue that our food supply is most at risk as it becomes increasingly a GMO monoculture. (And maybe the solution is as simple as taking measures to encourage genetic diversity.)
So i mean the consequences may be orders of magnitude smaller but I'd also argue the chance of something happening is orders of magnitude larger. (We're talking about small chances for either event, that's my point).
We've had genetic engineering as long as we've had farming. The difference is that we've applied industrial practice to everything. The result will be destructive in the long term, but profitable for now.
It's profitable because everything is fungible. There is no incentive to maintain the land. Ancient Rome literally had the same problem.
I think this is because while we've been messing around creating novel human genomes for at least two-hundred-thousand years, we've been creating non-human thought processes for only 40 years. And it's really only picked up in the last 10.
Our butts are at least partly covered that we do anything particularly awful on the genetic engineering front because genomes have been scrambling around in all kinds of different configurations for billions of years now and nothing too awful has happened. Except, arguably, us. Although I like to think we're the best result so far rather than the worst.
I don't buy the genetic engineering is natural argument for a couple of key reasons
The first thing is that with natural selection of cross breeding new strains start small and ramp up providing a natural provisional test period.
With genetic engineering the goal is often something like a blockbuster new type of corn which sees global adoption quickly. So if something unexpected happens there is a non zero risk of lots of people's food supplies not working out.
Further if we're making analogies to nature, it's important to note that terrible things happen in nature. The key part of natural selection is that only some organisms get to reproduce and the rest often live miserable short lives.
I mean, to be flippant, our butts are covered in the sense we probably won't see anything worse than zinka, sickle cell anemia or ebola?
While it's not difficult to imagine scenarios where we create bioweapons of unprecedented ferocity, state actors have probably already done this to some degree.
I argue that outside of engineered diseases it's difficult to imagine existential risks that could be posed by genetic engineering. Except, of course, making a better human which would almost certainly and immediately pose an existential risk to old Homo sapiens sapiens.
> While it's not difficult to imagine scenarios where we create bioweapons of unprecedented ferocity, state actors have probably already done this to some degree.
There is still a difference between "created" and "deployed at large scale".
Evolution is not a stochastic process, even without newer models like CRISPR where we know bacteria and other organisms seek very specific genetic material and integrate it into their own genome; evolution is governed by natural selection and your DNA has a lot of tricks under it's belt.
Your DNA was meticulously constructed by billions of years of evolution and that is not by chance since there are processes that construct DNA which are not based on "random mutation", you can take in foreign DNA, activate and deactivate genes put in spacers move base pairs from coding and non-coding DNA, modify gene expression and repetition and tons of other tricks and this all happens through a process not by chance.
If you think about evolution only in terms of random mutations which are filtered out through natural selection you throw away most of what we've learned about biology in the past 50 or so years, in fact random mutation is likely not have been workhorse of evolution especially in rapidly reproducing organisms.
Microorganisms edit their DNA and they do it at a very fast pace, this is why you can have bacteria developing resistance to drugs without even reproducing since they start seeking out resistive genes in their non-coding DNA and in their environment (plasmids and phages) as soon as they are exposed to antibiotics, they can integrate them into their coding DNA activate those genes and begin making the needed proteins and even rebuild their cell membranes if needed.
> imagine the code arose originally by random chance
If you said this in any other context, it would sound laughably ludicrous. Imagine billions of lines of code for an incredibly advanced, complex and self-replicating system appearing out of the thin air (over 2 bil. yrs) by mere chance. Even with the long time period, it sounds facetious, prima facie. For some reason people decide to switch off their common sense and rationality when it comes to this.
You don't understand how evolution works (not that the parent you're responding to stated it well). It is most certainly not "random chance".
Frankly I'm surprised to see that kind of sentiment coming out of such a CS-heavy audience, since genetic algorithms demonstrably work, and function on the same mathematical principles minus the biological substrate.
Don't basically all genetic algorithms rely on "random chance," or some simulation of random chance, to simulate a mutation rate?
And haven't biologists, in recent years, emphasized the degree to which evolution relies on mutations and genetic drift? That was my understanding of the way the field has been going, anyway.
Saying that evolution as a whole is just "random chance" seems reductionist, but randomness does appear to be involved.
Certainly randomness is _involved_, but it is randomness governed by a decidedly non-random fitness function. Without fitness there is no evolution.
The problem arises because when biologists, statisticians etc say "random process," they mean something very specific that can actually contain many different kinds of mathematical structure, but your average layman hears that term to mean "¯\_(ツ)_/¯"
Consider throwing 20 dice. It is certainly a random process. And we know the odds for throwing 20 sixes is about one to 30 000 trillion, so you'd have to keep throwing dice for about as long as the Earth has existed to get that result.
But if you start throwing the dice and you keep all the sixes you get along the way, Yahtzee-style, we expect you'll have 20 sixes in about 25-30 throws.
In the latter case, external selection has turned an incredibly implausible result into a certainty.
I don't agree with this analogy though. Who's to say if life evolved on earth all over again, we'd end up in the exact same place? Sure, some of the same adaptations would re-emerge. But some might not. And there might be extraneous mutations that had no effect on fitness at all but drastically changed the appearance of some subset of species. If we assume true randomness is involved then we assume evolution itself is going to deliver non-deterministic results, don't we?
To put it in terms of genetic algorithms, we could run through N generations M different times and get different results each time.
To me that makes it seem like saying "randomness is an important aspect of evolution" isn't so far from the mark.
We absolutely wouldn't end up in the same place, simply because the state space of possible organisms and environments is astronomically larger than the space of possible dice rolls (his metaphor isn't perfect and I'm already straining it here). But we would be almost guaranteed to end up with organisms that have similar levels of complexity.
Again, randomness is an important part of evolution, but there are also necessary deterministic aspects of it that make the "blind watchmaker" type arguments that started this discussion irrelevant.
The basic argument made by the "random chance" commenter was not that evolution is a random process, but that it favors organisms with complex and interdependent gene expression, because this increases the rate at which random mutations cause differentiation.
But if you start throwing the dice and you keep all the sixes you get along the way, Yahtzee-style, we expect you'll have 20 sixes in about 25-30 throws.
But this scenario has the implicit assumption of a deterministic outcome (all 6s), and also that after each throw, the outcome of chance is being analyzed according to a set of rules, which are being applied such that keeping/discarding dice is done to produce a deterministic outcome.
Even if you deny that there is traditional "intelligence" in the process such as a dice thrower, the scenario still fails to answer why there is even a set of rules for evaluating these independent dice throws in the first place, where these rules came from, and why those rules seemingly guide events of completely random chance in a pre-determined direction at all (all 6s).
The choice of a predetermined goal and an unchanging, deterministic selection function are just simplifying assumptions to make the analogy easier. Neither of those things is required for evolutionary processes to occur.
Arguably this is where the future of binary code is anyway: automatic static and dynamic analysis and simulation to automate patching. Machines will be much better at this than we can ever hope to be. One could imagine the only way to guarentee some level of saftey in genetic modification is via the same or similar means.
automatic static and dynamic analysis and simulation
[...] in genetic modification
That's going to be tough to do in humans. Mutations in the complement component 4 gene dramatically increase the risk of developing schizophrenia. The only way to test for that is to simulate a human growing from a fertilized egg, through childhood, into adolescence, to early adulthood, which is typically when schizophrenia manifests. You'll need to run a few thousand of these simulations, to have a sufficiently powered clinical trial.
And, of course, these simulations would have to equivalent to real humans, otherwise it'd be pointless.
Right. I certainly wouldn't argue that we do/don't have the computational power to do this or that it's even going to be possible as I'm nowhere near qualified. This is simply my layman's view of "bioinformatics"... I'd like to imagine we'd get something like this eventually.
There is one more problem. Schizophrenia developing is a part of development of human. Not just biological object, but real human, who use his neural system widely in a lot of developing tasks. So to be sure, you'll need to simulate human development. You'll need to simulate for a such experimental object complete environment with loving mother, child games, school, some social relatives (human is sufficiently social creature and he is unable to develop without social environment).
From technical point of view its just a computational complexity issue, but how about ethical point of view? How about human rights of simulated consious person?
If we replace degenerate alleles with the most-common version of that allele, isn't that pretty safe? It would have been tested on billions of people naturally.
The degeneracy of a given allele is relative to the phenotype of the organism and also the organism's environment. Sickle-cell confers survival advantage in environments where malaria is endemic, but is considered a degenerate allele in environments without malaria.
>Not if the people with the degenerate version also have another mutation that interacts.
Is this common? An uncommon detrimental variant that is slightly-mitigated by another uncommon detrimental variant, such that the removal of one makes the other worse isn't the type of thing I would expect to come about through natural selection.
Because we know genetic loci do not affect the phenotype in purely orthogonal ways (this is known as epistasis), there is a huge risk of having unforeseen second order consequences. In state space even on simple models like NK model for low K, epistasis can have profound effects on fitness metrics and survivability as organisms attempt to hill climb.
Maybe it's a matter of tooling. I don't want to appear disrespectful, but after having seen Christopher Domas present "The Future of RE: Dynamic Binary Visualization" my mind was forever blown. Working with hexdumps, debuggers, disassemblers etc. seems so stone-agey compared to that. Like so much in IT in general. Anyways, caveman CAN more than crawl, obviously ;-)
So, if you have up to an hour of time to waste, maybe you can have your mind blown to cantor dust too?
Also, there's an array of environmentally-determined switches and gene-determined proteins which are being changed concurrently by a team with whom you cannot communicate. Also the previous team which worked on all patches had several billion years and the specification was exacting.
Like trying to make a pretty patch on modular synth with access to only one module. This may be a bad idea.
Meh. Compilers generate pretty simple code, people have been patching complex binaries and ROMs for ages, with generally good and stable results. I think the genetic code generated by evolution is likely far more complex, convoluted and inter-dependent than any assembly spat out by a compiler, so I think the analogy goes too easy on genetic editing.
This isn't about editing genes, it is about replacing mitochondria. Not the same thing at all. Mitochondria are like bacteria in the way they replicate. There are hundreds of them in a cell. They can pass components around between one another, can split and fuse. They are integrated with quality control mechanisms that cull the herd. It is a complicated picture.
As the article notes the procedure isn't effectively clearing out all of the old mitochondria, and dynamics can favor them in the future.
It is known that variations in mitochondrial DNA produce radical differences in competition between mitochondrial strains in the cell. That is how deletions affecting OXPHOS machinery cause one mutant to take over the whole population very quickly - some differences produce mitochondria that either replicate better or resist quality control more effectively. That one is one of the causes of aging, but the same principle exists for other differences between mitochondrial genomes. If you put two or more in a cell and let them fight it out, hard to say in advance what the outcome will be given present knowledge.
So, basically, the people working on mitochondrial replacement need to make their tools for cleaning out the old mitochondria more efficient. If 100% success is achieved, that genome isn't coming back.
Alternatively, actual gene therapy might be a better approach - though challenging if you want to edit mitochondrial genomes, as you have the same problem of coverage and competition. There is allotopic expression, moving mitochondrial genes to the cell nucleus, which is feasible via today's gene therapy. Given the amount of work needed to copy mitochondrial genes into the nucleus, however, the challenge being how to alter them so as to get the proteins produced back to the mitochondria, something that has been achieved for three genes so far, it might be more cost-effective to work on better clearance and replacement technologies for the near term of assisted reproduction needs.
> Some research suggests that nuclear genes evolve to sync well with a mitochondrial haplotype, and that when the pairing is suddenly switched, health might be compromised.
Normally, eggs provide all mitochondria. The sperm's mitochondrion gets left outside during fertilization. So in most zygotes, there's a new pairing between the father's nuclear genes and the mother's mitochondrial haplotype.
I wonder what that entails. I do understand that most zygotes fail to implant, or get terminated very early in development. Maybe this is one of the causes. Anyone know?
The article is good but the headline is terrible, falsely implying some surprising new discovery rather than the sobering actuality that we lack a good theory of genetic development. A better one would be 'Gene Editing's Risks Are Hard to Manage.'
The nuclear/mitochondrial mismatch theory is interesting. Communities with a long history of admixture often have a significant number of individuals with mitochondrial DNA and nuclear DNA of divergent geographic origins, but I have not seen it observed that they are subject to any deleterious mtDNA/nucDNA 'mismatch' effects. I wonder what the threshold for this effect is.
Seeking out and using so called super mitochondria for this therapy could have it's own problems. If future generations only had such mitochondria they could become immune to future therapies. By selecting and actively promoting "healthy" mitochondria we could limit mitochondrial diversity which could lead to diseases which affect huge swathes of the population. Obviously we are not there yet but these are some of the dangers of applying gene therapy to broad populations.
This isn't gene editing, it's genome/organelle transfer. We have no science to explain the effects of organelle transfer. Also, none of these risks were "unexpected", except to people with very limited imaginations. Those of us who have followed advanced biomedicine for decades are rarely surprised when genomes behave "unexpectedly". They're not passive storage mechanisms made of just DNA that is accessed; they're complex, dynamic systems with a lot of behavior determined by the proteins that bind them.
Some days I feel like people should just go back and study Barbara Mcclintock until they "get it".
tl;dr: Earlier this month, a study published in Nature by Shoukhrat Mitalipov, head of the Center for Embryonic Cell and Gene Therapy at the Oregon Health and Science University in Portland, suggested that in roughly 15 percent of cases, the mitochondrial replacement could fail and allow fatal defects to return, or even increase a child's vulnerability to new ailments.
It's not a risk of editing the genes so much as a risk of the treatment failing.
Given that this is symbiotic relationship, this probably works in both ways but given that number of genes coded in mitos is low its probably way harder to notice the effect.
So it might happen that even if 100% of 'old mitochondria' is cleared the new symbiosis will also be defective as some of the 40 genes required by the hosting cell might be missing with new mito lineage replacing the original one.
What I've thought about is replacing the DNA in cells of adults with an average of multiple samples of their DNA, to correct for the mutations in our DNA that accumulates as we age. I think I've heard something about progress in this area, but I don't know what to google for. Does anyone know if there's any news in this area?
Genetic engineering has exactly the same problems that modifying binary-only code. After some very complex observation of the system you may spot the point that is responsible for a certain problem, but you will never be sure (as in provably sure) that the "fix" you put in place:
1. is going to stop all the occurrences of the behavior you want to fix;
2. will only affect the behavior you want to fix;
3. will have no effect on other systems that you did not modify.
(The article is mostly about 1 and a bit about 2 and 3.)
Modifying legacy code is hard enough when the uncommented source code is around. Changing directly the binary is both a great engineering feat and something to be scared of.