GD^2PO：通过群体动态奖励分离式策略优化来缓解多奖励冲突问题
GD^2PO: Mitigating Multi-Reward Conflicts via Group-Dynamic reward-Decoupled Policy Optimization
摘要
随着大语言模型的发展，训练后的强化学习越来越依赖于多维奖励来培养综合能力。这一转变要求新的算法能够同时优化多种可能存在冲突的目标。为了应对这一问题，现有的方法如“组式奖励分离策略优化”通过将整体得分分解为独立的奖励组，然后分别计算每个组的强化学习损失。然而，这种策略仍然面临多重奖励冲突的问题：同一个行动可能在某些奖励指标上带来积极效果，而在其他指标上则产生负面效果，导致相互冲突的信号在汇总过程中相互抵消，从而进一步降低强化学习的效率。受“动态采样策略优化”的启发，该方法通过过滤掉那些几乎没有优势且无效的行动来提高强化学习的效率。我们提出了“组式动态奖励分离策略优化”方法。具体来说，该算法采用一种基于冲突意识的过滤机制，来剔除那些在奖励方面存在严重矛盾的行动。通过这种方式，我们可以避免相互冲突的信号相互抵消，从而保持并增强有效强化学习优势的幅度，进而显著提升学习效率。此外，我们还引入了查询级权重调整机制，根据每个查询的整体奖励情况来动态调整其更新强度。在包括工具调用和人类偏好对齐等多种多奖励场景中的实验表明，GD2PO始终能显著优于现有的基线方法。代码可访问于https://github.com/Qwen-Applications/GD2PO。
English Abstract
As LLMs advance, post-training reinforcement learning (RL) increasingly relies on multi-dimensional rewards to cultivate comprehensive capabilities. This shift demands new algorithms capable of optimizing diverse and potentially competing objectives simultaneously. To address this, existing methods such as Group reward-Decoupled Policy Optimization (GDPO) decompose the overall score into independent reward groups, then compute the RL loss separately within each group. However, this strategy still encounters multi-reward conflicts: a single rollout can yield positive advantages on certain reward dimensions but negative ones on others, causing opposing signals to cancel each other out during aggregation, further hindering RL training efficiency. Inspired by Dynamic sAmpling Policy Optimization (DAPO), which improves RL training efficiency by filtering out ineffective rollouts with near-zero advantages, we propose Group-Dynamic reward-Decoupled Policy Optimization (GD^2PO). Specifically, GD^2PO employs a conflict-aware filtering mechanism to mask out rollouts suffering from severe reward-wise disagreement. By preventing conflicting signals from canceling each other out, this masking strategy preserves and enhances the magnitude of effective RL advantages, thereby significantly accelerating learning efficiency. Furthermore, we introduce query-level reweighting to dynamically adjust the update intensity of each query based on its overall reward consensus. Experiments on various multi-reward scenarios, including tool calling and human preference alignment, demonstrate that GD^2PO consistently and significantly outperforms existing baselines. The code is available at https://github.com/Qwen-Applications/GD2PO.