SPADE: Self-Play in Adaptive Synthetic Executable Environments

<p>This paper introduces <strong>SPADE</strong>, a reinforcement learning framework that enables a single large language model to achieve <strong>open-ended self-improvement</strong> by designing its own training worlds. One role, the <strong>Environment Designer</strong>, creates complex, multi-turn tasks as executable <strong>Python code</strong>, while the <strong>Reasoning Agent</strong> role learns to solve them. To ensure the tasks are challenging yet possible, the system utilizes a <strong>hint-based regret</strong> signal, rewarding the designer when an agent succeeds with a secret hint but fails without it. This competitive dynamic allows the training curriculum to <strong>automatically evolve</strong> in complexity as the model&#39;s capabilities grow. Research results demonstrate that <strong>SPADE</strong> significantly outperforms static training methods across various math, coding, and tool-use benchmarks. By turning environment creation into a <strong>learnable skill</strong>, the framework offers a scalable solution to the scarcity of high-quality human data.</p><p></p>