AI Extinction Is the Wrong Test

Last week, Evan Hubinger, who leads alignment science at Anthropic, said he believes artificial intelligence has a greater than 10 percent chance of killing every human within the next decade. His position makes the warning impossible to ignore: he works on the safety of Anthropic’s AI systems. His chief concern is that a future, more capable AI might escape human control.

Concern about that possibility has also reached Congress. Senator Bernie Sanders and Representative Greg Casar have announced a proposal to ban artificial superintelligence and pause advanced AI development while safety rules are established. A proposal of that scale calls for a precise account of the disaster it aims to prevent and how the proposed measures would help.

What would it take for AI to kill literally every human? A catastrophe would have to reach across the world, defeat attempts to contain it, and leave no surviving population. Conceivable routes include a biological disaster, nuclear war, or a prolonged collapse of systems essential to survival. AI might contribute through the actions of a person using it for harm, or through a future system acting beyond human control. Each possibility has a different sequence of events.

The first begins with a person. Major AI systems people use online, such as Claude and ChatGPT, are designed to refuse requests for help with cyberattacks or biological weapons. Those safeguards make misuse harder, though they can be evaded. The 2026 International AI Safety Report, chaired by Yoshua Bengio and written by an international group of experts, finds that safeguards are improving but remain imperfect. Providers can also monitor for misuse, giving a determined actor reason to seek another way to use AI.

Open weight models raise a deeper concern because people can download and run them privately. Someone can modify a model, remove its safeguards, and use it beyond a provider’s monitoring. Once released, its weights cannot be recalled. Even then, AI advice does not give someone the materials for a biological weapon or control over nuclear weapons. The question is how much the technology lowers the remaining barriers for a determined actor, and whether an attack could spread so far that no one survived it.

The second possibility is closer to Hubinger’s warning. A future AI might pursue a goal its creators did not intend, conceal what it is doing, and resist attempts to stop it. It would also need to carry out plans over time and gain access to reso