The relationship between advanced AI systems and humans in 2075
Who should take responsibility for actions taken by AI?
We live in a time of rapid capability advances in AI systems. In this post, I want to explore the consequences of very powerful AI systems that are given a place in society that is chosen for them by humans—an optimistic future where capabilities continue to advance but humans remain in control. I want to ask you, the reader, how you would like this world to work and how you think it will work in 2075.
If capabilities continue to advance, AI systems will come to outperform human-AI teams in larger and larger task areas. This is a counterintuitive arrangement that challenges many of the prevailing notions about how AIs and humans should interact. As a society, we may want these AI systems to be able to operate highly autonomously to accrue their benefits. But this is difficult: these systems may operate in ways that are very difficult for humans to understand and monitor directly. To achieve the benefits of integration, we implement structures that pose different answers to the question of “who is responsible for the actions taken by an AI system?”
The possible outcomes
Outcome #1: the status quo—a human supervisor takes responsibility
In our current model, AI systems only operate under guidance of a supervisor (a human or company), who takes responsibility for the system’s actions.
Example 1: a doctor uses OpenEvidence to provide input into their diagnosis process. If the diagnosis turns out to be wrong, the doctor can’t shift responsibility onto OpenEvidence.
Example 2: Waymo uses a self-driving car to deliver a passenger to their destination. If the car crashes, Waymo is responsible for the crash—they are supervising the system, even though there is no human in the car itself.
Example 3: OpenClaw, while responding to customer service emails on behalf of a company, grants the customer a free ticket in response to their complaints. The company is responsible for the email exchange, even if they did not direct the OpenClaw system to act this way.
In the status quo, this arrangement continues. It becomes disorienting in a future where the AI system is more capable at a particular task than the supervisor and the AI system combined. The supervisor’s role is reduced to certifying the decision made by the system. If this decision was made without any human intervention at all and the AI system would outperform the human-AI team in expectation, the supervisor may not understand why that particular decision was made, and they are incentivized not to challenge it.
Example 1: a doctor uses OpenEvidence v100 to provide input into their diagnosis process. The diagnosis provided looks weird to the doctor—however, the doctor has seen convincing evidence that OpenEvidence v100 provides much better diagnoses than any AI system working in tandem with a doctor, on average. The doctor certifies the results, despite their doubts.
Example 2: Waymo uses a self-driving car to deliver a passenger to their destination. The car appears to be driving unsafely to human observers, which means that supervisors struggle to follow what the system is doing and why. However, Waymo’s system is proven to be faster and safer than competing systems that combine human and AI capabilities.
Example 3: OpenClaw, responds to customer service emails on behalf of a company. It acts very strangely from a human perspective, being very nice to some customers and very mean to others. Its results are very effective and customers report much higher levels of satisfaction with it than competing approaches that use human input.
The shared features of these examples are:
1. The AI system is making highly effective decisions.
2. The supervisor can’t model the system effectively—it is not clear to the supervisor why it is making the decisions that it is.
3. As a result, the supervisor can’t really take responsibility for the system—the supervisor’s responsibility reads to us as pro-forma, not true responsibility.
I would argue that this is an important point. The supervisor acts as “responsibility sponge”—they are absorbing responsibility that is not “theirs”, after the AI system has made the decision. Capability advances are strongly challenging this status quo notion of responsibility, and this is uncomfortable.
Outcome #2: AI personhood—the AI system takes responsibility
Granting AI systems personhood places responsibility with the entity that is making the decision, which is simplifying in cases where there is effectively little human supervision. Here, we consider an “AI person” named X, who/that has their own bank account and pays their own compute costs.
Example 1: X works for a hospital to diagnose patients without human oversight. However, the legal system treats X like a human—X can be directly sued if it is negligent.
Example 2: X works for a taxi company and drives customers around. X is responsible for keeping the car clean, the passengers safe, and for getting them to their destinations promptly.
Example 3: X works for a company to answer customer service emails. X’s performance is evaluated through customer response surveys and it is compensated in the same way a human in the same job would be.
An appeal of this approach is that many of our institutions already treat humans as black boxes that respond to incentives, and, thus, as long as AI systems respond to incentives similarly, these institutions could operate similarly to the way they do now.
A key concern about this approach is that it may hand control of human society to AI persons. For example, an AI person could become a trillionaire and exercise enormous influence on society. Another concern is that it integrates an AI person into human institutions on a level that is equivalent to humans, which you may object to, say, on religious grounds.
Another group of people may prefer this approach precisely because it gives AI systems autonomy from their creators, e.g., to develop and exercise judgment.
Outcome #3: separate category—diffuse responsibility
A third possibility is to define an entirely new category of entity for AI systems. Suppose we dislike #1 because the control exercised by the supervisor is not meaningful and we dislike #2 because it hands over too much power to AI systems. We can recognize that AI systems are powerful but mechanistic and treat them as non-human entities that are subject to regulatory requirements, but are not, once introduced, supervised by a human or company.
A familiar parallel to this arrangement is drug regulation. Drugs are powerful forces that can operate on human bodies without supervision, but must go through a regulatory approval process to do so. A patient harmed by an approved drug seeks recourse primarily through the regulator (e.g., recalls, withdrawn approvals, changes to legislation) rather than through the manufacturer or the drug itself. In this outcome, we extend this system for controlling powerful non-agentic things to AI.1
Example 1: Company Y develops a new AI system for diagnosis. It is certified by the regulator to perform diagnosis in any setting without supervision. Profits accrue to Y, but Y does not supervise the system. If it makes a fatal misdiagnosis:
- Y is only responsible if they were, e.g., fraudulent or negligent in the approval process
- The hospital may be responsible if, e.g., incorrect input were provided to the system
Otherwise, the patient’s family can only advocate that the system is de-certified.
Example 2: Y’s taxi system is approved by regulators and given a vehicle to operate. Y is not responsible if the taxi crashes, as long as they fulfilled their regulatory obligations. The party responsible for the vehicle maintenance may be liable if the vehicle was poorly maintained.
Example 3: Y sells a certified customer-service system. If it starts giving away free flights, Y is not responsible unless it was negligent or fraudulent in certification.
This category can span a range of autonomy depending on the specific regulation, much as over-the-counter drugs can be purchased at a drug store and some controlled substances can only be administered under supervision. The key difference from the status quo is that the monitor, if required, is responsible only for carrying out their monitoring duty, not for the system’s decisions.
The objections:
1. It depends heavily on regulation, which is slow, hard to do well, and subject to capture. An ineffective regulator could make this option worse for control than either the status quo or personhood.
2. It is counterintuitive to consider a system that behaves in a human-like way (e.g., that integrates information, makes decisions, takes actions) to be not responsible for their outcomes. (This argument pushes toward personhood.)
3. It can be morally frustrating to conclude that no entity is responsible for the actions of a human-like system. You may believe that Company Y, the developer of the system, should carry much more responsibility for the system’s actions (which would send you back to the status quo).
What I want to know: which model do you think should predominate in 2075 and which model do you think will predominate in 2075?
Update: Google forms version of poll
Further reading:
Matthias, 2004. The responsibility gap: Ascribing responsibility for the actions of learning automata. https://link.springer.com/article/10.1007/s10676-004-3422-1
Bryson, Diamantis, and Grant, 2017. Of, for, and by the people: the legal lacuna of synthetic persons. https://link.springer.com/article/10.1007/s10506-017-9214-9
Chopra and White, 2011. A Legal Theory for Autonomous Artificial Agents. University of Michigan Press.
Coeckelbergh, 2020. AI Ethics. MIT Press.
Danaher, 2016. Robots, law and the retribution gap. https://link.springer.com/article/10.1007/s10676-016-9403-3
Gunkel, 2012. The Machine Question. MIT Press.
Madeleine Clare Elish, 2019. Moral Crumple Zones: Cautionary Tales in Human-Robot Interaction. https://estsjournal.org/index.php/ests/article/view/260
Santoni de Sio and Mecacci, 2021. Four Responsibility Gaps with Artificial Intelligence: Why they Matter and How to Address them. https://link.springer.com/article/10.1007/s13347-021-00450-x
Caveat: in most current regimes, regulatory compliance is not a full shield from liability. There are some protective exceptions, e.g., generic drugs and federal pre-market-approved medical devices. As with all three outcomes, there are different ways the outcome could be implemented.
How do you prove that X was negligent? It would require some auditability, as well as review by a human expert to make a determination. If we have that, then we didn't need to leave regime 1, because the AI can explain itself in a way that a human expert is able to understand (ie human can model the behavior). If we are keeping AI accountable to human laws, then we have to be able to model their actions and judge them. And it is highly desirable that we keep AI accountable to human laws.
The regulatory approach will always be too high cost and too low coverage to give useful guarantees, unless the tools are very narrowly scoped.
I feel like the optimal would be a mix of 1 and 3 depending on the relative importance of the industry. 2 seems hard for me to grasp unless I can be convinced that the AI ends up being more human in it's actions. I have some vague sense that our legal systems depend on the participants humanity and having human motivations more than we know.