一种基于梯度优化的RLVR稳定性分析与胜者优势策略优化方法
A Gradient Perspective on RLVR Stability and Winner Advantage Policy Optimization
摘要
基于可验证奖励的强化学习（RLVR）能够提升语言模型的推理能力，不过类似GRPO的优化方法仍然容易陷入不稳定状态。我们通过分析令牌级别的梯度变化来研究这种不稳定性，并提出了一种分类体系，该体系可以预测各种更新对后续令牌的概率和熵的影响。该分类体系表明，系统的稳定性取决于当前策略下的优势函数以及令牌分布状况。基于这一发现，我们提出了“胜者优势策略优化”算法（WAPO）。这是一种简单的在线策略梯度优化方法，它只针对具有正优势的完成动作进行更新。在数学推理和多跳问答等基准测试中，WAPO显著提升了训练的稳定性，其性能甚至优于多种模型框架中的传统方法。完整的代码可以在https://github.com/layer6ai-labs/wapo下载。
English Abstract
Reinforcement learning with verifiable rewards (RLVR) improves language-model reasoning, but GRPO-style optimization remains prone to collapse. We analyse this instability through token-level gradient dynamics, deriving a taxonomy that predicts how updates affect next-token probabilities and entropy. The taxonomy shows that stability depends jointly on the advantage sign and token distribution under the current policy. Motivated by this finding, we propose Winner Advantage Policy Optimization (WAPO), a simple online clipped policy-gradient objective that updates only on positive-advantage completions. Across mathematical reasoning and multi-hop QA benchmarks, WAPO improves training stability and matches or outperforms baselines across multiple model families. Full code can be found at https://github.com/layer6ai-labs/wapo.