The Great Debate: Bounded Refinement vs Open-Ended Self-Improvement

This episode surveys "Recursive Self-Improvement in AI," a paper by Mingguang Chen and colleagues that classifies 1,250 papers on how AI systems attempt to improve themselves. The discussion establishes precise definitions distinguishing agents, harnesses, and evaluators, then draws a critical line between bounded self-refinement (improvement against a fixed external evaluator) and open-ended recursive self-improvement (where the system also modifies its own criteria for success). It traces the intellectual lineage from I.J. Good's 1965 "intelligence explosion" concept through Schmidhuber's provably-optimal but practically unusable Gödel machines, showing how the field traded mathematical proof for empirically checkable but weaker signals like benchmarks and tests. The episode also introduces a verification hierarchy ranking formal verifiers above execution feedback, learned judges, and self-assessment, citing key findings that scoring reasoning steps beats scoring final answers, and that language models largely cannot self-correct without external feedback. Listeners interested in AI safety, agent architectures, or the theoretical limits of self-improving systems will find this a rigorous framework for cutting through loose talk about "self-improving AI."

Sources:

1. Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loops — Mingguang Chen, Licheng Wang, Bo Qu, 2026

http://arxiv.org/abs/2607.07663

2. Speculations Concerning the First Ultraintelligent Machine — I. J. Good, 1965

https://scholar.google.com/scholar?q=Speculations+Concerning+the+First+Ultraintelligent+Machine

3. Gödel Machines: Self-Referential Universal Problem Solvers Making Provably Optimal Self-Improvements — Jürgen Schmidhuber, 2003 (extended 2006)

https://scholar.google.com/scholar?q=G%C3%B6del+Machines%3A+Self-Referential+Universal+Problem+Solvers+Making+Provably+Optimal+Self-Improvements

4. Self-Rewarding Language Models — Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, Jason Weston (Meta AI/NYU), 2024

https://scholar.google.com/scholar?q=Self-Rewarding+Language+Models

5. Darwin Gödel Machine: Open-Ended Evolution of Self-Improving Agents — Jenny Zhang, Shengran Hu, Cong Lu, Robert Lange, Jeff Clune (Sakana AI / UBC), 2025

https://scholar.google.com/scholar?q=Darwin+G%C3%B6del+Machine%3A+Open-Ended+Evolution+of+Self-Improving+Agents