We don’t know the difference between AI risk and uncertainty and it shows

I’ve expressed concern about our over-reliance on direct, authoritative responses from chatbots. There’s no space to go back and forth with a chatbot; you ask a question — which doesn’t necessarily have an obvious answer — and a chatbot will respond with confidence, collapsing any uncertainty or ambiguity that may have been in the query.

Interestingly, there’s now a study that shows us what would happen if we embedded expressions of uncertainty (e.g. “I’m not sure, but…”) into AI chatbots:

• When an AI chatbot offers a response, people tend to agree with it.

• But — and this is an important ‘but’ — expressions of uncertainty decrease agreement and confidence in the chatbot’s outputs. So if the chatbot starts its response with “I’m not sure, but…” or “It’s not clear, but…” people are less likely to agree wholesale with whatever follows — and they’re less likely to feel confident in the output.

• Expressions of uncertainty don’t impact whether participants anthropomorphize the chatbot.

So, sure, let’s add the phrase “I’m not sure, but…” to the beginning of every chatbot response. But let’s not confuse this tactic with a concerted effort to manage risk — this is just a warning label. Indeed, our entire conversation about ‘AI risk’ confuses risk with uncertainty. Risk is the management of an unknowable thing; whereas uncertainty is when you can’t manage that unknowable thing. As Vaughn Tan writes,

“True uncertainty is defined by the unquantifiability and unmanageability of not-knowing (as opposed to risk, which is not-knowing which can be quantified and thus managed and eliminated).”

When we map this distinction against the current AI discourse, it becomes clear real fast just how confused the conversation is. For example, we talk about ‘alignment’ in terms of risk — AI safety experts are hired to align the AI systems developed by companies with human values. But as I’ve argued in the past, alignment isn’t a problem to be solved, it’s a myth. The belief in ‘alignment’ misunderstands how humans can hold conflicting values that cannot be optimized away. Alignment isn’t a risk companies can manage, it’s true uncertainty hiding in plain sight.

We talk about ‘hallucination’ as a risk or potential harm to manage and mitigate. But as I’ve argued before, ‘hallucinate’ is the wrong word because “it implies an aberration; a mistake of some kind, as if it isn’t supposed to make things up.” That’s exactly what these models do — given a bunch of words, the model probabilistically makes up the next word in that sequence. It’s not something we can quantifiably manage or control. Sure, the tech can improve, but it will only ever offer a probabilistic guess based on existing data sources — again, uncertainty grows while hidden from view.

Uncertainty is also alive in how the chatbots separate information from its context. Professor Emily Bender makes this point clear with a medical query. A traditional search engine, Bender explains, would return several links, including the Mayo Clinic, WebMD, Dr. Oz’s site, and a Reddit-like forum where people ask similar questions. As Bender contends, a chatbot removes the medical information from its context. You don’t know whether the response came from the Mayo Clinic or Dr. Oz. As Bender writes:

If instead of the 10 blue links, you get an answer from a chatbot that reproduces information from some or all of these four options (let's assume it even does so reliably), you've lost the ability to situate the information in its context.

When we separate information from its context, we lose the ability to manage what’s misunderstood or confusing. Ignoring context is how we end up with the rock-eating and pizza-gluing recommendations, as that information once made sense in a sub-Reddit, which Google decontextualized and used as training data.

This decontextualization isn’t something we can manage by adding citations. It turns out, AI systems are terrible at citing their sources, and this may well be a feature, not a bug. As Matteo Wong writes in The Atlantic:

Some of the problems may be inherent to the setup: Reliable summary and attribution require adhering closely to sources, but the magic of generative AI is that it synthesizes and associates information in unexpected ways. A good chatbot and a good web index, in other words, could be fundamentally at odds—media companies might be asking OpenAI to build a product that sacrifices “intelligence” for fidelity. “What we want to do with the generation goes against that attribution-and-provenance part, so you have to make a choice,” Chirag Shah, an AI and internet-search expert at the University of Washington, told me.

This all reminds me of agnotology or the study of “how or why we don’t know.” It’s like a cousin of epistemology, which asks “how we know.” Anyway, at its core, agnotology is the idea that “ignorance should not be viewed as a simple omission or gap, but rather as an active production,” according to Robert Proctor, a historian of science. You can think of censorship, propaganda, marketing campaigns, email deletion policies, and non-disclosure agreements all as creating a kind of ignorance.

The production of ignorance and not-knowing is rife in an information ecosystem increasingly mediated by AI. As a result, we mistake probabilistic guesses for ‘hallucinations.’ We mistake reflections in a mirror for a prediction about the future. We mistake uncertainty for risk we can manage. We mistake the production of ignorance for intelligence. And so, we look for solutions in all the wrong places, such as in warning labels and citations. These small and unimaginative interventions do nothing to change our actions and challenge fundamental incentives and norms; a disclaimer warning you that ChatGPT is uncertain will never be as powerful as building AI systems — if we want to build them at all — in tandem with new social systems that improve our quality of life.