Apr 20, 2026·Formal Sciences·260421.0054
Too Correct to Learn: Reinforcement Learning on Saturated Reasoning Data
Zhenwen Liang, Yujun Zhou, Sidi Lu, Xiangliang Zhang +2 more
This paper addresses a critical problem in reinforcement learning for large language models: when base models are already very accurate on training benchmarks, standard RL methods fail because there aren't enough errors to learn from, causing models to collapse into repetitive solutions. The authors propose CUTS, a novel sampling strategy that maintains solution diversity even when models are highly accurate, improving generalization on challenging out-of-domain math problems by up to 15.1%.