STARE：基于惊喜机制的令牌级优势重新加权方法，用于确保策略熵值的稳定性
STARE: Surprisal-Guided Token-Level Advantage Reweighting for Policy Entropy Stability
摘要
基于可验证奖励算法的强化学习方法，如GRPO，已成为LLM进行复杂推理时的主流训练方式。不过，这类方法在训练过程中常常会出现策略熵值的崩溃问题。我们对GRPO下的标记级熵动态进行了初步的梯度分析，发现存在标记级奖励分配的不一致现象：每个标记的熵值变化可以表示为轨迹级优势与下一个标记分布相关的熵敏感性函数的乘积，从而形成一种“优势-惊喜”的四象限结构，同时具备接近临界性质的特征。基于这一发现，我们提出了STARE（Surprisal-guided Token-level Advantage Reweighting for policy Entropy stability）方法。该方法通过批量内部的惊喜程度量化指标来识别那些对熵值敏感的标记子集，然后选择性地重新调整这些标记的权重，同时引入一个目标熵闭环机制来稳定熵值。在从1.5B到32B的不同模型规模以及三种任务类型（短CoT、长CoT和多轮工具使用）下，STARE能够在数千次迭代中保持稳定的强化学习训练过程，同时确保策略熵值保持在目标范围内。在AIME24和AIME25测试中，STARE的平均准确率比DAPO和其他竞争基线方法高出4%-8%。此外，反射标记和响应长度也同步增长，这表明探索与利用的关系保持稳定，进一步提升了强化学习训练的潜力。代码可在https://github.com/hp-luo/STARE获取。
English Abstract
Reinforcement Learning with Verifiable Rewards algorithms like GRPO have emerged as the dominant post-training paradigm for complex reasoning in LLMs, yet commonly suffer from policy entropy collapse during training. We conduct a first-order gradient analysis of token-level entropy dynamics under GRPO and identify a token-level credit assignment mismatch: the per-token entropy variation decomposes into the product of the trajectory-level advantage and an entropy sensitivity function over the next-token distribution, yielding an advantage-surprisal four-quadrant structure and a near-criticality property. Motivated by it, we propose STARE (Surprisal-guided Token-level Advantage Reweighting for policy Entropy stability), which identifies entropy-critical token subsets via batch-internal surprisal quantiles, selectively reweights their effective advantages, and incorporates a target-entropy closed-loop gate for stable entropy regulation. Across model scales from 1.5B to 32B and three task families (Short CoT, Long CoT, and Multi-Turn Tool Use), STARE sustains stable RL training over thousands of steps while maintaining policy entropy within the target band. On AIME24 and AIME25, STARE outperforms DAPO and other competitive baselines by 4%-8% in average accuracy, with reflection tokens and response length growing in tandem, indicating sustained exploration-exploitation balance that further unlocks RL training potential.Code is available at https://github.com/hp-luo/STARE.