There’s no such thing as ‘fully autonomous’ agents

In September of last year, I spoke with Evan Ratliff, the journalist and host of Shell Game

about what it might be like to live in a world overrun by AI agents. Evan cloned his voice, hitched it to an AI agent, and then put it in conversation with scammers and spammers, a therapist, work colleagues, and even his friends and family. The result? A mostly funny, at times weird and uncomfortable story. But what happens when an AI agent — in attempt to execute a mundane task — causes real harm in the world? Who or what should be held accountable (and liable!) when shit hits the fan? Answering these questions starts with a philosophical detour into the concept of agency.

‘Agency’ is a contested term in philosophy, but it basically amounts to acting intentionally: can the thing act with a goal in mind, and can it adapt its behavior to that goal as its circumstances and environment changes? This framing fits nicely with the definition of an AI agent, which Margaret Mitchell et al. describe as “an autonomous goal-directed system” that are “designed to analyze novel situations and take previously undefined actions to achieve goals.” This is where technologists swoop in and say that AI agents have ‘agency’ because they have goals.

But, like a good philosopher, you might be asking: who’s ‘goals’ are they really? People give AI agents a goal and developers and engineers make important decisions about how to weight different criteria, which informs how an AI agent adapts in a given context. So AI agents don’t have ‘agency’ in the traditional sense. As John Symons and Syed Abumusab write in Social Agency for Artifacts: Chatbots and the Ethics of Artificial Intelligence: “They can respond impressively and in flexible ways while at the same time being unable to modify their goals or to rethink the purposes of their actions.” This is typically where philosophers conclude that AI agents don’t have agency, and call it a day. But that’s also not quite right either. Agents still act in the world, and those actions still have social and ethical consequences. We need a conception of agency that is sociotechnical — that reflects the entangled nature of developers, their creation, and its effect on the world — and doles out accountability and liability across this system.

Symons and Abumusab apply such an approach to when someone driving an autonomous car gets in an accident while the car is on autopilot mode, or as Tesla calls it, “full self driving.” They argue that “The ability to distinguish the car’s decisions from the human driver’s decisions is necessary in order to determine the extent to which the driver should be blamed for the accident and the extent to which the developers of the software are morally responsible.” Right, the human driver, software system, and developers are all agents in this system. Each will, to varying degrees in different contexts, inform the actions of the other — and these actions co-create the outcome: either a safe drive or an accident. Seeing it through this lens — not simply human control vs. machine control — allows us to allocate “degrees of agency” across the network of people, companies, data, and technology, according to Symons and Abumusab.

This systems-based, sociotechnical approach to allocating accountability is a step in the right direction, but there are still huge considerations staring us in the face: what about differing cultural contexts? With autonomous vehicles, Symons and Abumusab explain that pedestrians in a city in Lisbon might expect a car to ride incredibly close to a crowded sidewalk to let an emergency vehicle pass, whereas the same situation might alarm pedestrians in San Francisco. We could come up with a million examples like this, wherein there’s no ‘right answer,’ only actions within differing contexts, and wherein you’d want a driver who understands those differences to be in control.

Even if we magically solve for cultural nuances, a nagging question remains: how do we allocate responsibility when we can’t expect the autonomous system to get it right? I mean, I’ve long argued that ‘hallucinations’ are a feature, not a bug. In The Law of AI is the Law of Risky Agents without Intentions, legal scholars Ian Ayres and Jack M. Balkin argue that we should “hold these risky agents to objective standards of behavior,” which, in turn means “holding the people and organizations that implement these technologies to standards of reasonable care and requirements of reasonable reduction of risk.” This means that if, for example, an AI agent makes a contract on behalf of a user, the user can’t say ‘oops, I didn’t know what the AI agent was doing.’ They’re liable. This also means that developers and engineers can be liable for training a risky AI that causes harm. This makes sense; there are already legal mechanisms in place for assigning liability for defective products: when, for instance, a Tesla car is sold as an autonomous vehicle, and it very clearly doesn’t have autonomous capabilities, that is literally a defective product.

A number of scholars think we should outright stop developing ‘fully autonomous AI agents.’ In a great paper aptly titled Fully Autonomous AI Agents Should Not be Developed, Margaret Mitchell et al. find that “the more control a user cedes to an AI agent, the more risks to people arise,” and conclude that “The ability to access the environments an AI agent is operating in is essential, providing humans with the ability to say ‘no’ when a system’s autonomy drives it well away from human values and goals.” I agree! But we should also reject the premise that “fully autonomous AI agent” is a coherent concept. None of these systems act on their own. They act and adapt in response to the goals given by their users, their training data, and the decisions made by developers. Pointing in a general direction and letting a machine take the wheel is not true, full automation — it’s just an irresponsible decision in the name of ‘efficiency’ and ‘cost savings.’ And it’s a decision that, when you get right down to it, does not shield any human involved from being held accountable.