Theoretical Limits of Language Model Alignment

<p>This paper investigate the <strong>fundamental limits of language model alignment</strong> by establishing exact theoretical boundaries for reward improvement under a KL-divergence constraint using <strong>Jeffreys divergence</strong> and a computable <strong>covariance estimator</strong>. They demonstrate that <strong>best-of-N sampling</strong> closely approaches this theoretical Pareto frontier, whereas gradient-based methods like <strong>PPO and GRPO</strong> remain suboptimal. Additionally, the literature analyzes how <strong>proxy reward errors</strong> drive performance degradation and <strong>reward hacking</strong>, while proving that <strong>reward ensembling</strong>successfully mitigates these issues at a convergence rate of <strong>O(n⁻¹ᐟ²)</strong>.</p>