Is the AI Race Out of Control?

The people building the world’s most powerful artificial intelligence systems promise extraordinary gains. They also warn those systems could escape our control.

That is the ethical fault line running through this week’s episode of The Business of Tech.

My guest, New Zealander Peter Barnett, works at the Machine Intelligence Research Institute in Berkeley, California. The University of Otago physics graduate develops governance plans aimed at preventing catastrophic outcomes from advanced AI. He previously worked on technical AI alignment, the difficult job of making powerful systems act in line with human intentions.

This conversation is a topical one.

OpenAI is facing fresh turmoil over its approach to safety. Three recently fired safety and alignment researchers claim they were punished for challenging the company and working with outside evaluators. OpenAI rejects that account, saying they mishandled sensitive information and breached its trust.

David Robinson, one of OpenAI’s senior safety figures, has also resigned. The former author of safety reports accompanying major model launches says the company’s culture is “broken” and the industry is not being nearly careful enough. He just sat down for a podcast interview with Ezra Klein of the New York Times, which makes for fascinating viewing.

These departures expose the moral dilemma at the centre of the AI boom.

Worrying emergent traits in AI

The companies racing towards more capable models say the technology could cure disease, lift productivity and create enormous wealth. But they face fierce competition, huge investor expectations and a geopolitical contest framed around beating China.

Every incentive rewards speed. Caution risks handing the advantage to a rival.

So who decides when the danger is too great?

Barnett argues it cannot be the companies themselves. He says frontier AI developers are effectively writing their own rules and can loosen them when safety commitments obstruct progress. Independent audits, government oversight and limits on training more powerful models may all be needed.