Discussion about this post

User's avatar
Jack's avatar

Semi structured notes:

Re: variance as one of the four conditions. Assuming we've solved catastrophic forgetting.

Reliance on hardware that isn't biological -- AI might copy and persist but the grass will outcompete the data centers at some point

Lack of bottleneck is hard. This is where aggressive regulation can help. Biotech/pharma already does this via the FDA and the broader biochemical threat enforcement infrastructure. Instantaneous replication and the general anonymity frameworks available across the web work against this heavily... but "dangerous" models are still extremely large (I agree with your para that covers most of this, ending with citation to pacing discourse). There's also a dimension here where a "better aligned" AI can compete on Behalf of the humans, in a way nuclear and bio don't have strong analogues.

Scenario 1 Medium term is one of the scarier parts of the thought experiment, because of the 21st century evidence. We have plenty of examples to suggest that rights would be sacrificed to ensure safety. Though it's hard to imagine a world where personal compute is genuinely restricted... the economy runs on serving ads via personal compute. Maybe aggressive agent vs human verification steps in here?

"Moral status of AI systems" i don't feel anything when I read this statement, but I wonder if others do or if a system in the next few years would make a compelling case for us to consider it.

It feels like there are analogies here to viruses that are "too aggressive". They might be ultra effective at escaping containment and causing damage but that objective can work against persistence, if they're too lethal. AI need long term infrastructure to really accomplish anything... for now and the foreseeable future that requires significant human activity to run the supply chains that generate power and operate compute infra.

"I see no value in this outcome either, because these are not values I care about." Indeed. lol.

Re: will of the universe arguments. It's annoyingly easy to convince yourself that there is no universal objective function other than survival and selection and so everything that's ever happened has just contributed to the next stage of selection. If it's compute then it's compute. However... two things. 1) compute is reaaaallllyy disadvantaged in terms of its ability to sustain itself on earth. There's just such a massive chain of things that have to be built for AI as it exists today to be able to sustain itself physically. 2) I have an evolutionary drive to ensure that humans are never outcompeted so I find counter arguments annoyingly *uncompelling*. Arguments on consciousness and worthiness seem unidentifiable... I'm aligned with you that some evaluation of the values that we can ascribe to decision making seems to be the "right" question. It certainly covers more space than just considering the end state (ie we're judging all observable SA pairs, not just S_H)

Emergence of altruistic values... recent work showing small tweaks in post training can lead to broad changes in willingness to cheat, etc. AI competing with each other though is something else than human training tweaks... some other global optima/limit argument ... seems like starting point induced by last human training point still matters (uncomfortable stochasticity)

Section 3: it's just hard to imagine comparing a human to a non-biological human. We're so intimately defined by our biological hardware, and consciousness, as far as we know, is an emergent feature of that hardware. Unless we copy those features completely in compute, I don't know how valid any comparison would even be. Said another way, why would we be more "human" than "AI", if we copied someone's brain onto compute?

Human-compute hybrid systems feel easier to compare... at least as long as they are augmenting core human systems. But the line gets blurred quickly here doesn't it depending on which part of the system is "driving".

"A deep technical understanding of the human brain is required in this scenario" -- indeed.

Para on periods of vulnerability in humans is so interesting, and reflection of post-training/RL removing whatever it might have adopted from text about our emotional/evolutionary attachment to protection. In some sense that might work against the AI though wouldn't it. If it devalues "stopping existence" it should also devalue the desire to persist and compete.

"gradient of similarity, and an identity anchored that way may carry values that are unrecognizable to us." -- uncomfortable mix of convincing and difficult to reason about.

"allow more time for human values to transfer" that's a really interesting sentence because it's counter to how I thought about the competition for the rest of the essay (ai and human are separate systems independently optimizing), but here it assumes we can still influence the AI as it competes with us, even if we decide to let it establish after an escape.

"Unique to this scenario is an understanding of how moral values evolve under pressure, which is needed to shape the circumstances of competition." Sounds like a nice series of papers that someone should write ;)

Votes:

Should: humans remain in control (pesky evolutionary drive to outcompete)

Will: of the scenarios posted... probably 4. Escape seems likely. Challenges of silicon sustaining itself will give us lots of time to figure out how to shape competition. 3 seems likely to take much longer than any of these other scenarios... and 2 seems unlikely given its requirement that regulation basically doesn't happen.

Ready for more?