μ子何时有助于代理强化学习？
When Does Muon Help Agentic Reinforcement Learning?
摘要
在大规模预训练方面，Muon与AdamW具有竞争力，但其在强化学习后训练阶段的性能表现仍不明确。我们使用Qwen2.5-0.5B-Instruct框架，在ALFWorld上通过与AdamW的单一种子对比实验来研究vanilla Muon在稀疏奖励代理式强化学习中的表现。在Group-in-Group Policy Optimization（GiGPO）机制下，仅对隐藏权重矩阵应用Muon后，最终窗口验证成功率从0.290提升到了0.546，即增加了88%；而高频率使用的AdamW则没有这样的提升效果。这种效果取决于优势估计器以及学习率的取值。当学习率为3e-5时，Muon使得GRPO的成功率从0.161提升到0.268；而在GraphGPO中，晚期窗口的差距则随着饱和现象的出现而缩小。当学习率为1e-5时，GraphGPO的Muon模型则达到了0.901的准确率，标准化验证AUC也从0.399提升到了0.556；同时，在30次和60次更新后，其成功率分别达到了0.5和0.75。这些探索性结果表明，Muon能够为代理式强化学习带来积极效果，因此有必要进一步研究策略优化器、优势估计器以及学习率之间的相互作用。多种子和跨任务验证工作仍有待进一步开展。
English Abstract
Muon is competitive with AdamW in large-scale pre-training, but its value for reinforcement-learning (RL) post-training remains unclear. We study vanilla Muon in sparse-reward agentic RL through matched single-seed comparisons with AdamW on ALFWorld using Qwen2.5-0.5B-Instruct. Under Group-in-Group Policy Optimization (GiGPO), applying Muon only to hidden weight matrices raises final-window validation success from 0.290 to 0.546 (+88%); high-rate AdamW controls retain no post-update success. The effect depends on the advantage estimator and learning rate. At 3e-5, Muon improves GRPO from 0.161 to 0.268, whereas GraphGPO's late-window gap narrows near saturation. At 1e-5, GraphGPO Muon reaches 0.901, raises normalized validation AUC from 0.399 to 0.556, and reaches 0.5 and 0.75 success 30 and 60 updates earlier, respectively. These exploratory results show that Muon can benefit agentic RL and motivate studying the policy optimizer, advantage estimator, and learning rate jointly. Multi-seed and cross-task validation remain open.