Episode #193 – The Locus of Intelligence

The intelligent party in a conversation with a frontier model is not the model. Both the people building these systems and the people warning against them assume otherwise, and that shared assumption is the same error the nineteenth century made about gold once the telegraph arrived: mistaking the part of the work a new technology does for the whole of it. Intelligence turns out to be a property of a loop rather than of a system, and the loop only produces knowledge when something inside it pays the constraint.

Episode Summary

Almost everyone who argues about artificial intelligence agrees on one thing without noticing they agree on it: that the intelligent party in a conversation with a frontier model is the model. The people building these systems believe it. The people demanding that the building stop believe it too. When two camps that agree on nothing else converge on a premise, the premise is the part worth examining, and this one is wrong. Intelligence is not a property the system in your chat window has on its own. It’s a property of a loop, and the loop has to contain something that pays a particular price.

The mistake has a precedent, and it’s a monetary one. Lyn Alden has made the point about gold and the telegraph: before the 1840s, information about gold travelled at roughly the speed of gold itself, because both moved on ships. A ledger entry in London claiming that a thousand ounces had landed in Boston could not honestly be written until a ship reached Boston and someone confirmed the metal was sitting there. Communication ran at settlement pace. The telegraph cut the two apart. Information about gold could now cross an ocean in minutes while the gold itself stayed at ship speed, which meant ledgers could assert things that had not been verified, and nobody downstream could tell the verified claims from the unverified ones. What followed was fractional reserve expansion, paper layered on paper, and a hundred and seventy years later, 1971.

The same mistake is now being made about knowledge. A frontier model does part of the work of producing knowledge, the part where a plausible sentence about something gets written down. People have started to act as though it does all of the work, including the part where the sentence is true, or where there is good reason to think it is. Those are different jobs. The telegraph did the communication half of settlement and got mistaken for the whole of it; the model does the generation half of knowledge and gets mistaken for the whole of it. What differs this time is the clock. Gold’s version of the error took a hundred and seventy years to pay out. This one has maybe twenty-four to thirty-six months before the consequences start to bite.

Look at what the model does at the smallest unit, a single sentence. Ask it about the causes of the French Revolution and it runs one mechanical operation: predicting which word is likely to follow the last one, given everything in its training data. It has no model of the French Revolution, only a model of what sentences about the French Revolution tend to look like. Most of the time that does not bite, because the sentences come out clean. A clean sentence carries a signature, though. Every sentence in the training corpus was written by a person who, to arrive at it, threw away a pile of alternatives that were wrong or off-topic or not quite what they meant, and the survivor carries the mark of that throwing-away. The model reproduces the mark and skips the work. It’s Maxwell’s demon run backwards: a process that looks like it is doing what only constraint-paying labour can do, while the constraint itself goes unpaid, and it holds up only because somebody already paid that constraint once, in the training data.

Step back from the single sentence and the same severance shows up at the scale of an entire knowledge system. For nearly all of history,