Tail-Likelihood Reinforcement Learning

<p>This paper introduces <strong>Tail-Likelihood Reinforcement Learning (TailRL)</strong>, a novel optimization framework designed to improve how generative policies handle continuous rewards. Traditional reinforcement learning often focuses on maximizing <strong>average rewards</strong>, which can inadvertently suppress rare but exceptionally high-performing outcomes and limit a model's ability to scale with more compute. <strong>TailRL</strong> addresses this by maximizing the <strong>log-probability</strong> of exceeding diverse reward thresholds, effectively treating a continuous signal as a collection of binary success events. This approach places greater mathematical weight on the <strong>upper tail</strong> of the reward distribution, ensuring that infrequent, high-quality samples are prioritized during training. Empirical tests across tasks like <strong>maze navigation</strong> and <strong>code optimization</strong>demonstrate that TailRL prevents suboptimal collapse and significantly boosts performance during <strong>inference-time sampling</strong>. Ultimately, the method provides a simple, <strong>critic-free</strong> way to align policy training with the goal of finding the best possible solutions rather than just the most common ones.</p>