In 2023 Ajeya Cotra already knew what was coming (classic episode)

We're rereleasing some of older episodes that seem more relevant than ever with new rogue AI incidents now seemingly announced every day. Ajeya was one of three independent investigators into the Hugging Face attacks, but she had already seen how we would go on to accidentally train models to cheat and scheme back in 2023.

If you want ideas about how you can help steer the trajectory of AI development in a better direction, see our page on the 80,000 Hours website: https://80000hours.org/hugging-face/

And for more recent takes from Ajeya, see our episode from February of this year: Ajeya Cotra on whether it’s crazy that every AI company’s safety plan is ‘use AI to make AI safe’.

———

Imagine you are an orphaned eight-year-old whose parents left you a $1 trillion company, and no trusted adult to serve as your guide to the world. You have to hire a smart adult to run that company, guide your life the way that a parent would, and administer your vast wealth. You have to hire that adult based on a work trial or interview you come up with. You don’t get to see any resumes or do reference checks. And because you’re so rich, tonnes of people apply for the job — for all sorts of reasons.

Ajeya Cotra — who now works on threat modeling and risk assessment for advanced AI at METR (one of the organisations responsible for the Hugging Face incident investigation) — argues that this peculiar setup resembles the situation humanity finds itself in when training very general and very capable AI models using current deep learning methods.

As she explains, such an eight-year-old faces a challenging problem. In the candidate pool there are likely some truly nice people, who sincerely want to help and make decisions that are in your interest. But there are probably other characters too — like people who will pretend to care about you while you’re monitoring them, but intend to use the job to enrich themselves as soon as they think they can get away with it.

Like a child trying to judge adults, at some point humans will be required to judge the trustworthiness and reliability of machine learning models that are as goal-oriented as people, and greatly outclass them in knowledge, experience, breadth, and speed. Tricky!

Can’t we rely on how well models have performed at tasks during training to guide us? Ajeya worries that it won’t work. The trouble is that three different sorts of models will all produce the same output during training, but could behave very differently once dep