One Word Changed Every Answer: Ben Wills on Testing LLMs Directly

Ben Wills ran the same hotel-recommendation prompt over and over and changed exactly one word in it: the car he was flying to Los Angeles to pick up. Honda Civic, BMW, Ferrari. The hotels moved with the car, and they moved across every ChatGPT model and every level of thinking he tried.

Ben started in SEO in 2001 at KeywordRanking alongside Andy Beal and Garrett French, co-founded the link building company ontolo with Garrett, then disappeared into 13 or 14 years of pure engineering. Large-scale crawlers and scrapers written in C by hand. Embedded firmware for an ESP32 inside high-end audio equipment. He is back in marketing now, building AI search visibility tooling at OppAlerts, and his argument is that LLMs finally hand SEOs the thing search engines never did: a fast, direct, testable feedback loop.

In this episode

The one-word experiment: hold the prompt constant, swap the car, and watch the hotel tier move with it

Why you can test an LLM but never could test a search engine: no waiting for the index, no personalization to strip out

The persona is a ranking factor: 100 industries, 10 personas each, 1,100 personas across the July analysis

Backlinks, ranking strength and LLM visibility: a decent correlation, with a full page of caveats in front of it

How Ben actually uses a correlation: as the anecdotal evidence that tells you which experiment to run

Isolating the model: thinking off, fan-out off, then turn each one up a step at a time

AI slop as an accountability question: producer standards versus consumer appetite, and whether his own reports qualify

Inside OppAlerts: historical PageRank from Common Crawl, news-specific link gap analysis, 20,000+ taxonomy categories

The Yandex source leak: a week in a Slack channel with Mike King and Russ Jones, and what the category vectors showed