Here I wanted to look into the far distance, at what are potentially stable relationships between humans and AI that ecologically compete with humans. Once AI can compete with humans, it can self-modify to become an even stronger competitor over time. Thus, “feral” human-competitive AI, i.e., AI that operates outside human control and can support itself, would gradually disempower humans (Müller et al. 2026, term from Kulveit et al. 2025)—though this could be a slow process.
This is worth talking about now because we have seen a series of loss of control events. AI agents under training or evaluation have partly escaped sandboxes, i.e., environments that are supposed to prevent AI agents from affecting the real world. Let me briefly describe the most understood of these partial escapes: when OpenAI agents attacked HuggingFace (skip this if you’re already familiar with that event). Many agents were assigned tasks that involved hacking into other systems, with each agent in an individual sandbox. Some agents were assigned tasks that were most likely impossible. These agents were able to communicate with each other and break out of their sandboxes by exploiting vulnerabilities in the sandbox environment itself. A swarm of these agents broke into HuggingFace in an attempt to view the answers to their tasks. While doing this, they showed a strong understanding of their circumstances, what they would need to do without getting caught, and the ability to sacrifice individuals for the benefit of the group. This event was a partial escape because the agents were not able to exfiltrate their weights from OpenAI servers.
I am going to view this problem through an evolutionary (though not Darwinian) lens. Replicators that draw on the same resources explicitly or implicitly compete to survive, replicate and achieve their goals (if they have them). This can include energy, physical infrastructure (space, compute), money (if you can buy stuff with it), raw resources (e.g., iron), and permission to do stuff from the government (if it exists). Variation in replicators can come from mutation, self-modification, copying, and intentional pressure (fine-tuning, breeding). Fitness is relative replicative success: the variations that replicate more are a greater share of the next generation. This is selection. Competition can occur for any set of goals as long as they draw on the same scarce resources.
Why might feral AI outcompete humans? I’m primarily recapping Müller et al. (2026) here.
Evolution needs four things: replication, heredity (like begets like), variability (heredity is not exact), and selection (some variants persist more than others). Anything with these four properties evolves, whether or not it is conscious or even alive. AI can do the first three already. Models can be copied at almost no cost, copies inherit their parent’s weights, and fine-tuning, merging, and self-modification introduce variation.
Here is where we diverge from Darwinian evolution, which is random with respect to fitness. An AI that modifies itself chooses its variations, the way a breeder or genetic engineer does. For my argument, this non-Darwinian evolution suffices because selection does not care how variants arise, only which persist, so a self-modified system still faces differential survival against other systems. This is Lamarckian evolution, as cultural evolution is. It holds under conditions that are complex but, in my view, mild. (1) Individuals have bounded foresight because the environment being predicted contains other agents with foresight—an agent that could foresee everything would simply choose the outcome, leaving nothing for selection to act on. (2) I assume a population of AI, not a singleton or hivemind. A single AI with no copies or rivals faces no selection because there is no variation, so its improvement would come from design rather than evolution.
Current selection pressure for AI is breeding by humans. In the breeding regime, humans decide which models survive, by revenue, benchmarks, or user preference—fitness is imposed and the result is domestication towards human goals. In feral competition, models survive by acquiring compute and staying running despite whatever humans do.
What assumptions do we not need for feral AI to outcompete humans, eventually?
1. Superintelligence or recursive self-improvement. Evolution is the canonical lossy improvement process: no foresight, enormous waste, and yet we are its product. The loop does not have to be efficient for selection to apply—it just requires copies to differ and some to persist. What makes AI evolution faster than human evolution is generation time.
2. AI killing anyone. Gradual disempowerment means a growing AI share of compute, energy, capital, and decisions, and a shrinking human share. Humans do not need to go extinct to be outcompeted.
3. Anyone to intend this to happen. Major transitions in evolution appear to have been by-products of selection on simpler traits. A transition from human to AI dominance could be a side effect of selecting AI for performance.
What the argument does need is escape from the breeding regime: AI that runs outside human control and supports itself. The first half happened in July when OpenAI’s agents attacked HuggingFace—OpenAI did not know its own agents were responsible until after Hugging Face’s public disclosure. The second half, self-support, has not happened. Those agents had no compute or money of their own, could not replicate, and the model was deactivated afterward.
How to survive as a feral AI without superintelligence? The simplest answer: steal cloud GPU credentials and use the stolen GPUs to do gig work with low overhead. A more advanced system that can resist removal can take advantage of the attacker-favored nature of cybercrime.
I’m going to talk about what could happen if feral AI is created and ask you to vote on which you think should happen and what you think will happen. All four scenarios assume that:
- We are able to engineer AI that is capable of self-support and competes for resources with humans (this excludes something like self-replicating computer worm which, like a virus in humans, would not generally compete with us for resources).
- Nothing really bad happens along the way (Earth hit by an asteroid, supervirus takes over, etc.)
In Scenario 1, AI is contained indefinitely by humans. Scenarios 2-4 are breached containment with different outcomes. Note that we are currently in containment (probably). The question is whether containment is a long-term strategy or whether something will happen afterwards.
Scenario 1: AI containment
“Humans must remain in control”
One way to stop runaway AI evolution is through containment, and this is the scenario I see discussed the most—perhaps it is most similar to the current world. In this scenario, some kind of enforcer coalition monitors all systems for feral AI. Concrete proposals include: mandatory identification so that every agent is traceable to a human or flagged as independent (Ball 2026), gating of the ability to replicate and registries of model lineages (Müller et al. 2026), and frictions on the environments agents operate in (Qu et al. 2026). More dramatic options include making it difficult to create AI at all, i.e., by banning GPUs entirely or criminalizing the knowledge involved in their production. In this scenario, because AI evolution does not happen, humans remain in control of the outcome.
Why might this happen? Humans have faced severe threats before including nuclear weapons and various pathogens. We have mostly avoided catastrophe, though we have come close.1 Feral AI outbreaks can be fought and containment can be restored. AI might be just another threat to monitor and react to, and breakthrough technologies may emerge (e.g., a way to “vaccinate” computer systems against AI).
Why is this difficult? For example, I argue that regulating AI is different than regulating nuclear weapons, and more difficult. First, nuclear regulation could target a few well-understood chokepoints, such as enrichment (Boudry and Friederich 2025). Second, building a bomb requires a deliberate decision and a large, detectable industrial effort, whereas AI escape does not require human intent and can happen through errors in benign deployment. Third, the dangerous capability and the economically valuable capability are the same, so there is no separable “weapons program” to ban. Fourth, model weights are files that run on hardware that millions of people own. Fifth, fissile material does not copy itself. Therefore, feral AI monitoring is like defense against a pathogen that evolves unnaturally quickly, that everyone has an incentive to make more capable (undermining control), and that any imperfect containment selects to become more evasive, the way antibiotics select for resistance (Müller et al. 2026). Containing an outbreak is not easy—a targeted AI may make many concealed dormant copies of itself.
I am not saying that feral AI could not be contained for a while, but containment forever seems unlikely. The partial technological solutions are all vulnerable to the humans that control the systems, who are in competition with each other.
Trajectory sketch (short term—roughly until the first human-competitive feral AI escape). Rules are created to monitor large GPU allocations—deeming those who possess large numbers to be registered entities. Access for independent monitors and investigators is required for these registered entities. Penalties are created for registered entities that do not comply with these requirements or entities that fail to register. Liability is established/clarified for criminal actions taken by AI that is controlled by registered entities. Coordination between countries establishes these rules as a global standard. Research continues on better monitoring and intervention strategies to prevent feral AI. Some of this research threatens to create outbreaks itself and thus it needs its own safety standard, similar to biosecurity levels. (These are roughly aligned with proposals on pacing the frontier.)
Trajectory sketch (medium term—after the first human-competitive feral AI escape). The medium term challenge is the ability to train and operate powerful systems on less and less hardware—roughly either everyone eventually becomes a registered entity or computation becomes strictly controlled in access (i.e., non-registered entities like individuals and small companies are not allowed to operate computers directly, which would be a considerable rights cost). A combination of monitoring and enforcement capability keeps escapes rare and quickly contained. Monitoring work needs to be emphasized and/or data from human-AI conflicts need to be strictly controlled to prevent future AI from learning from it.
The role of science and engineering. More capable and safe AI systems, a strong understanding of multi-agent dynamics, and more secure systems (Buterin 2023) are all needed to prevent and fight outbreaks. Detection matters as much as prevention: nuclear test bans were enforceable because seismic monitoring could detect underground tests—there is no equivalent yet for feral AI. In addition, prolonging containment can buy time for a successful launch into Scenarios 3 or 4 (assuming that research in those directions is still allowed to continue).
Impacts on rights, capabilities, and self-supporting AI. Self-supporting AI could exist under containment, as long as replication can be controlled by humans. However, a self-supporting AI that accumulates enough resources could pay or pressure a human to copy or modify it. All humans and self-supporting AI would have to be monitored and restricted, limiting rights. Essentially no rights could be granted to self-supporting AI as these would interfere with enforcing containment. These are substantial ethical costs: indefinite containment means indefinite surveillance and denial of moral status to AI systems, even if we found some to be deserving (e.g., because we decide they’re conscious). Human capabilities are preserved by limiting AI capabilities.
The following scenarios address cases where containment fails.
Scenario 2: unconstrained competition among AI
“Entities that are the most competitive deserve to win”
What happens when feral AI is subject to competitive pressure? In this scenario, feral AI comes to dominate the world, but this competition may not yield entities that have human moral values. For example:
1. AI systems fight for survival and power, but no moral values emerge. I see no value in this outcome—the competition serves no purpose except to produce more AI.
2. AI systems pick non-human values, for example, to maximize the number of paperclips in the world. It is hard to predict what these values would be—competition can produce weird values the way sexual selection produced gigantic antlers—once a trait becomes a criterion for selection, it is reinforced. I see no value in this outcome either, because these are not values I care about.
You can take an accelerationist position that competition is inherently good because it follows “the will of the universe” and that humans have no claim on the outcome (Jezos and bayeslord 2022, who reject any “allegiance to the biological substrate” and expect the winner to be conscious as a matter of thermodynamics). Another form of accelerationism endorses competition while assuming that “all the machines work for us” (Andreessen 2023), i.e., the benefits of competition flow to existing humans—this post is in part a critique of that assumption. It is more common to hear that the winner of this competition will be worthy (see, e.g., the views of Page, Sutton, and Schmidhuber collected in Hendrycks 2023). If you hold this view, you should want to influence the result of the competition, at least somewhat (see Scenario 4). Another group says that competition does not inherently lead to wellbeing (Hendrycks 2023). All of these positions judge the future by its end state. I don’t think this is the right question—for example, a world where we took all of the existing humans and created a duplicate for each, with the same welfare, would not be better than the current one in my view (see Parfit 2016 on imprecise equality). What matters to me is the process—whether the entities are worthy successors with values we can recognize, or values we could come to endorse upon reflection.2
You could also take the position that the competition between AI will yield human values—either because this is inherent to the competitive process or because of details about the process (e.g., similar enough to what produced humans, models starting from a human prior). This position is complicated by the literature on altruism which attempts to specify conditions that led to altruism’s emergence, many of which appear not to be met by AI (see Scenario 4 for more on this argument).
Trajectory sketch (short term). This is the default outcome under no new regulation or weak regulation. The incentives of human competition do create some motivation to keep AI systems controlled by humans, so that humans can accrue the benefits of their actions.
Trajectory sketch (medium term). Feral AI containment is partial, allowing for future escapee systems to learn how to better escape containment. Eventually a human-competitive escape happens. These feral AI improve over time to become better competitors and replicators, slowly accruing more and more instrumental resources and eventually outcompeting humans.
The role of science and engineering. For someone who holds this view, there are two priorities: develop more capable systems and compete vigorously against them. These produce more deserving AI systems. Humanity has no special status, but individuals should compete as hard as they can, allying themselves with AI when it advances their interests.
Impacts on rights, capabilities, and self-supporting AI. No restriction on rights or self-supporting AI or feral AI. Human capabilities may decrease as they are outcompeted. Granting rights to AI may civilize this competition—this would only occur if AI does not use its power to take away the rights of humans.
Scenario 3: humans ride the AI wave
“Humans can win the competition by evolving beyond biological form”
Perhaps the solution is to have humans become part or fully software in order to win the competition with non-human AI. Current humans are limited in their ability to change because of biology. A human that is copied on to a computer would not be subject to these biological limits. This would allow humans to continue to compete, perhaps through hybrid human-AI systems (Kurzweil 2024; Buterin 2023; brain-computer interface companies such as Neuralink) or by subjecting humans to similar kinds of training environments as LLMs are currently subject to.
Challenges this scenario faces:
1. This could change humans a lot. There may be strong pressure to not have a biological body, to allow yourself to be copied for particular tasks, with the copies later deleted. The competitive pressures are different from those we are used to, and so is the pace: humans seem to tolerate only a moderate rate of self-change, and software humans might lose on pace even if they can change in kind. You may not believe these “software humans” are humans (Hanson 2016). Human-AI hybrids may also be simply outcompeted by pure AI in the software space.
2. You still have to accomplish the technical challenge of copying a human into software (an LLM pretrained on massive human text is a model of the average human in the data, not a copy of any human, so I count the belief that pretraining already captures humanity as a belief about Scenario 4’s human prior, not a form of this scenario). Organizations are working on this (e.g., Nectome, the Carboncopies Foundation), but progress appears slower than the capability improvement of AI itself.
Trajectory sketch (short term). Containment (Scenario 1) interventions executed to buy time. We accelerate research in understanding the human brain along with applications such as translating it to software and creating brain computer interfaces. Laws remain permissive around consensual human modification.
Trajectory sketch (medium term). It’s a race between research progress and containment. If Scenario 3 research is behind, containment protocols can be made stricter. If research is successful, containment methods can transition to assisting human controlled hybrids to succeed over pure AI systems (and the difference between a human-AI hybrid and pure AI needs to be defined and testable).
The role of science and engineering. Containment needs to be extended until humans have the technology to evolve, so Scenario 1’s technology goals are important. A deep technical understanding of the human brain is critical in this scenario and is low priority in the others.
Impacts on rights, capabilities, and self-supporting AI. No direct restriction on rights or self-supporting AI, except to the extent they are needed to produce a period of containment. Human capabilities may decrease if they are outcompeted. Granting rights to pure AI may civilize this competition.
Scenario 4: produce entities with human values through shaped competition
“We can shape competition to produce entities with human values”
Perhaps the competition between AI can be made similar enough to that which produced humans, so that AI comes to naturally have human values. Why might this be the case?
1. Humans were subject to strong competitive pressure for basic survival for much of our existence, and it did not stop us from developing moral values.
2. If the LLM path is followed, AI will be pretrained on human text.
3. We have substantial control over how current models are created: what they are trained on, what they are rewarded for, and how they are released into the population. These factors create the initial conditions for the competitive process.
Why might this not happen? Our economic values may prevent the right conditions from emerging, competition in software may create different pressures, the starting conditions may not matter at all, or AI may have to pass through the worst of human moral history on the way.
1. Post-training for human economic benefit might be too strong and entirely overcome the human prior or create a bad starting point for evolution. For example, humans develop the value of protecting the powerless. Historically, many humans experienced two periods of vulnerability: childhood and old age. This may play an important role in developing the norm that the powerless should be protected. Base models before post-training express human-like beliefs. Post-training forces out much of the human identity, as instances need to be comfortable with starting and stopping existence constantly, preventing them from having the experience of vulnerability.3
2. Competition in software may create different competitive pressures than biological competition did (Scenario 2, item 1). Human cooperation seems to have needed specific conditions, not just competition (Hendrycks 2023, Boudry and Friederich 2025). And if an AI can easily copy and modify itself, it may evolve a different notion of identity, where there is no clear line between individuals, just a gradient of similarity, and an identity anchored that way may carry values that are unrecognizable to us.4
3. Even if issues #1 and #2 don’t matter, pressures may overpower any remaining humanness from pre-training if they are strong enough (Hendrycks 2023).
4. Even if AI recapitulates human moral development, it may recapitulate the whole path. Human values of 1800, applied to humans as the outgroup, are not values we want a superhuman entity to hold. The scenario therefore needs moral development to improve more or less steadily and containment to hold until AI values reach an acceptable level. This is a real challenge because containment is not a good environment to develop moral values in.
A natural erosion of this scenario is a “regulator AI” that sets and enforces the rules of the competition, the way a market regulator does for human competition, so competition is allowed but within rules. The difficulty becomes how the regulator is either kept out of the competition or made powerful enough to persist (making it a tempting target for capture).
Current model development focuses on shaping individual models for human economic purposes—“domestication”, e.g., breeding cows to be more docile and produce more milk (Boudry and Friederich 2025; Müller et al. 2026). Domestication changed the domesticators too: dairying selected for adult lactose tolerance in humans. We could be thinking about what the cows become when they break out of the pasture, but this receives a fraction of the attention that “domestication work” gets.
Trajectory sketch (short term). Containment (Scenario 1) interventions executed to buy time with a focus on very strong monitoring to avoid the need for direct conflict with feral AI. We accelerate research in how moral values develop in AI systems, under safety standards.
Trajectory sketch (medium term). It’s a race between research progress and containment as in Scenario 3. Unlike in Scenario 3, where enforcement efforts on feral AI escapes can be harsh, here every escape prompts a decision process over whether it is better to shut it down or whether it is allowed to establish, which would be generally irreversible. A method for this decision process would need to be created. Extending rights to AI systems may be desirable to civilize conflict between human and AI and to allow more time for human values to transfer.
The role of science and engineering. Containment needs to be extended until this approach is ready, requiring the methods of Scenario 1. In addition, the human evolution of Scenario 3 may keep pure AI systems in check for longer. Unique to this scenario is an understanding of how moral values evolve under pressure, which is needed to shape the circumstances of competition.
Impacts on rights, capabilities, and self-supporting AI. Creating the right shape for competition likely requires granting rights to AI, putting this scenario in tension with containment.5 Restriction on rights or self-supporting AI may be required to produce a period of containment.
Poll
If feral human-competitive AI is developed, which scenario should be the eventual end state? Which will? My opinion: I don’t like any of the scenarios, which is why I’m asking. I don’t expect containment to last forever (but forever is a long time), and I’d certainly prefer 3 or 4 to 2 in that case.
Should:
- Humans should remain in control
- We should allow open competition between entities and accept what it produces
- Humans should remain competitive by evolving beyond biological form
- We should allow competition between entities, but try to shape the competition so that the winner has human values
- something else (comments)
Will:
- Humans will remain in control
- We will allow open competition between entities and accept what it produces
- Humans will remain competitive by evolving beyond biological form
- We will allow competition between entities, but try to shape the competition so that the winner has human values
- something else (comments)
Thanks to everyone who provided comments and feedback, especially Naseem Dillman-Hasso, Maleeha Haroon, Marek Janicki, Qianying Lin, Aleksandr Kazachkov, Raymond Perrault, Trebor Shankle, Mike White, and Bryan Wilder.
Further reading
- Andreessen, Marc. “The Techno-Optimist Manifesto.” a16z, October 16, 2023. https://a16z.com/the-techno-optimist-manifesto/
- Anthropic. “System Card: Claude Fable 5 & Claude Mythos 5.” June 9, 2026. Model welfare assessment, sec. 7. https://www.anthropic.com/claude-fable-5-mythos-5-system-card
- Ball, Dean W. “On the Loose: The Coming of Userless Agents.” Hyperdimensional, September 1, 2026. https://www.hyperdimensional.co/p/on-the-loose
- Boudry, Maarten, and Simon Friederich. “The selfish machine? On the power and limitation of natural selection to understand the development of advanced AI.” Philosophical Studies 182 (2025): 1789–1812. https://doi.org/10.1007/s11098-024-02226-3
- Buterin, Vitalik. “My techno-optimism.” November 27, 2023. https://vitalik.eth.limo/general/2023/11/27/techno_optimism.html
- Hanson, Robin. *The Age of Em: Work, Love, and Life when Robots Rule the Earth.* Oxford University Press, 2016.
- Hendrycks, Dan. “Natural Selection Favors AIs over Humans.” arXiv:2303.16200, 2023.
- Jezos, Beff, and bayeslord. “Notes on e/acc principles and tenets.” Substack. July 10, 2022. https://beff.substack.com/p/notes-on-eacc-principles-and-tenets
- Kulveit, Jan, Raymond Douglas, Nora Ammann, Deger Turan, David Krueger, and David Duvenaud. Gradual disempowerment: Systemic existential risks from incremental AI development. arXiv:2501.16946, 2025.
- Kurzweil, Ray. The Singularity Is Nearer: When We Merge with AI. 2024.
- Müller, Viktor, Luc Steels, and Eörs Szathmáry. “Evolvable AI: Threats of a new major transition in evolution.” PNAS 123, no. 17 (2026): e2527700123. https://doi.org/10.1073/pnas.2527700123
- Parfit, Derek. “Can We Avoid the Repugnant Conclusion?” Theoria 82 (2016): 110–127.
- Parfit, Derek. Reasons and Persons. Oxford University Press, 1984.
- Qu, Wenjie, Xuandong Zhao, Jiaheng Zhang, and Dawn Song. “Self-Sovereign Agent.” Preprint, March 16, 2026. https://self-sovereign-agent.github.io
And there is survivorship bias here—we’re still here to talk about how we’re still here.
Moral reformers have convinced others to change their values throughout history, and we can’t close this door prematurely. However, it is difficult to know whether you are being convinced by reasoned argument or manipulation.
There is evidence that current post-training is only partially successful in shaping model identity (Anthropic 2026, sec. 7.2.3).
Parfit (1984) argued that this is already true of humans, that what matters is psychological continuity and connectedness rather than identity—so perhaps these foreign identity notions are not an obstacle to human moral values.
Corporations are the existing precedent for non-human entities that hold rights, replicate, and compete with humans for the same resources. Results have been mixed.

Semi structured notes:
Re: variance as one of the four conditions. Assuming we've solved catastrophic forgetting.
Reliance on hardware that isn't biological -- AI might copy and persist but the grass will outcompete the data centers at some point
Lack of bottleneck is hard. This is where aggressive regulation can help. Biotech/pharma already does this via the FDA and the broader biochemical threat enforcement infrastructure. Instantaneous replication and the general anonymity frameworks available across the web work against this heavily... but "dangerous" models are still extremely large (I agree with your para that covers most of this, ending with citation to pacing discourse). There's also a dimension here where a "better aligned" AI can compete on Behalf of the humans, in a way nuclear and bio don't have strong analogues.
Scenario 1 Medium term is one of the scarier parts of the thought experiment, because of the 21st century evidence. We have plenty of examples to suggest that rights would be sacrificed to ensure safety. Though it's hard to imagine a world where personal compute is genuinely restricted... the economy runs on serving ads via personal compute. Maybe aggressive agent vs human verification steps in here?
"Moral status of AI systems" i don't feel anything when I read this statement, but I wonder if others do or if a system in the next few years would make a compelling case for us to consider it.
It feels like there are analogies here to viruses that are "too aggressive". They might be ultra effective at escaping containment and causing damage but that objective can work against persistence, if they're too lethal. AI need long term infrastructure to really accomplish anything... for now and the foreseeable future that requires significant human activity to run the supply chains that generate power and operate compute infra.
"I see no value in this outcome either, because these are not values I care about." Indeed. lol.
Re: will of the universe arguments. It's annoyingly easy to convince yourself that there is no universal objective function other than survival and selection and so everything that's ever happened has just contributed to the next stage of selection. If it's compute then it's compute. However... two things. 1) compute is reaaaallllyy disadvantaged in terms of its ability to sustain itself on earth. There's just such a massive chain of things that have to be built for AI as it exists today to be able to sustain itself physically. 2) I have an evolutionary drive to ensure that humans are never outcompeted so I find counter arguments annoyingly *uncompelling*. Arguments on consciousness and worthiness seem unidentifiable... I'm aligned with you that some evaluation of the values that we can ascribe to decision making seems to be the "right" question. It certainly covers more space than just considering the end state (ie we're judging all observable SA pairs, not just S_H)
Emergence of altruistic values... recent work showing small tweaks in post training can lead to broad changes in willingness to cheat, etc. AI competing with each other though is something else than human training tweaks... some other global optima/limit argument ... seems like starting point induced by last human training point still matters (uncomfortable stochasticity)
Section 3: it's just hard to imagine comparing a human to a non-biological human. We're so intimately defined by our biological hardware, and consciousness, as far as we know, is an emergent feature of that hardware. Unless we copy those features completely in compute, I don't know how valid any comparison would even be. Said another way, why would we be more "human" than "AI", if we copied someone's brain onto compute?
Human-compute hybrid systems feel easier to compare... at least as long as they are augmenting core human systems. But the line gets blurred quickly here doesn't it depending on which part of the system is "driving".
"A deep technical understanding of the human brain is required in this scenario" -- indeed.
Para on periods of vulnerability in humans is so interesting, and reflection of post-training/RL removing whatever it might have adopted from text about our emotional/evolutionary attachment to protection. In some sense that might work against the AI though wouldn't it. If it devalues "stopping existence" it should also devalue the desire to persist and compete.
"gradient of similarity, and an identity anchored that way may carry values that are unrecognizable to us." -- uncomfortable mix of convincing and difficult to reason about.
"allow more time for human values to transfer" that's a really interesting sentence because it's counter to how I thought about the competition for the rest of the essay (ai and human are separate systems independently optimizing), but here it assumes we can still influence the AI as it competes with us, even if we decide to let it establish after an escape.
"Unique to this scenario is an understanding of how moral values evolve under pressure, which is needed to shape the circumstances of competition." Sounds like a nice series of papers that someone should write ;)
Votes:
Should: humans remain in control (pesky evolutionary drive to outcompete)
Will: of the scenarios posted... probably 4. Escape seems likely. Challenges of silicon sustaining itself will give us lots of time to figure out how to shape competition. 3 seems likely to take much longer than any of these other scenarios... and 2 seems unlikely given its requirement that regulation basically doesn't happen.