超越熵：基于正确性意识的优势塑造机制——通过对比策略优化实现
Beyond Entropy: Correctness-Aware Advantage Shaping via Contrastive Policy Optimization
摘要
基于可验证奖励的强化学习（RLVR）通常使用熵来构建优势函数。然而，熵无法区分有用的不确定性与有害的混乱状态，因此作为正确性信号的效力有限。我们提出了对比策略优化方法（CPO），该方法利用参考引导生成分布与普通生成分布之间的标记级差异来构建优势函数，从而确保正确性。理论和实证结果都表明，这种差异能够可靠地反映标记级的正确性。我们还发现，在线策略蒸馏是CPO的一种特殊情况，此时后验分布由外部教师模型生成。CPO还能解决零优势问题。在域内和域外测试集上的实验表明，CPO在保持良好泛化能力的同时，其性能明显优于基于熵的RLVR方法。进一步分析表明，正确和错误的响应分别有助于探索与利用，平衡两者才能获得最佳性能。
English Abstract
Reinforcement learning with verifiable rewards (RLVR) commonly uses entropy for advantage shaping. However, entropy cannot distinguish useful uncertainty from detrimental confusion, limiting its effectiveness as a correctness signal. We propose Contrastive Policy Optimization (CPO), which uses token-level contrastive disagreement between reference-guided and vanilla generation distributions for correctness-aware advantage shaping. Both theoretical and empirical results show that this disagreement reliably indicates token-level correctness. We further show that On-policy Distillation is a special case of CPO, where the posterior distribution is instantiated by an external teacher model. CPO also resolves the zero-advantage problem. Experiments on in-domain and out-of-domain benchmarks demonstrate that CPO substantially outperforms entropy-based RLVR methods while maintaining strong generalization. Further analysis shows that correct and incorrect responses naturally support exploration and exploitation respectively, and balancing both leads to the best performance.