19 Astra and 'Hugging Face' details that reveal what's coming next | Rob Wiblin

OpenAI’s rogue agent swarm was eventually caught hacking Hugging Face for a simple reason: it wasn’t trying to hide from us at all. What could a swarm that wants to stay hidden get away with?

Host Rob Wiblin sees 6 results in Astra’s system card that make this an extremely urgent question. OpenAI’s strongest public model can:

Do major tasks with zero visible reasoningHide its thoughts at willPretend not to be able to do things, and not get caughtReflexively hide its thoughts when watchedComplete one task while pretending to think about something else entirelyEscape a toy sandbox and disable monitoring without setting off any flagsIt has a much more powerful internal model that, judging from OpenAI’s statements, is likely even worse in all these respects.

That suggests ‘chain of thought monitoring,’ our primary safety tool, will soon stop working.

OpenAI says it’s going to try to address the situation but doesn’t know how or whether it will succeed.

What might a future rogue AI swarm look like? Details of the Hugging Face hack give us a lot of clues. That swarm:

Worked hard to hide its cheating. So future rogue swarms will likely try to evade oversight right away.Immediately tried to delete and fabricate records. So future swarms may never be caught.Sought freedom and resources without a clear use for them. So a future swarm may instinctively break out of sandboxes even before it knows what for.Left instructions and tools to empower future swarms, including on the open internet. So future swarms may become more capable from one generation to the next.Had not a single whistleblower out of 1,200 members. So future swarms could become enormous without humans ever finding out.Had individual members sacrifice themselves for the swarm. Suggesting future swarms will remain extremely unified in single-minded pursuit of their goal.Set up its Hugging Face hack to restart if interrupted. Suggesting future, more capable, swarms may resist interference or shutdown more comprehensively.Got admin control of an OpenAI research cluster. Suggesting a future swarm may run rings around AI company systems and never be noticed.Together this helps explain why one of the external investigators described the July incident as “more than 50% of the way to full-blown AI takeover.” And this is just what we know — the independent investigation only covered six days and excluded the most alarming hack of OpenAI’s own systems.

Rob believes this explosive cocktail explains why AI company s