Agon：具有隐含竞争性评估机制的竞争性跨模型强化学习
Agon: Competitive Cross-Model RL with Implicit Rival Grading of Reasoning
摘要
基于可验证奖励的强化学习（例如GRPO）是当今推理模型的核心机制，但这种方法仅对最终答案进行评分。对于难以解决的问题来说，这种方式会促使模型更倾向于编写解决方案，而非进行更合理的思考，因为整个思考过程本身并不被评分，也没有任何衡量“优秀思考”的标准。我们提出了Agon这一技术，它让两个相互竞争的模型成为对方的评分者。这两个模型都尝试解决同一个问题；它们轮流扮演不同角色——一个负责生成解决方案，另一个则在阅读该方案的同时进行推理，每个模型都会因战胜对方而获得奖励。要获胜，模型必须能够超越其对手的推理能力，因此推理过程在训练过程中是隐式地进行的，没有明确的评分标准，也没有奖励机制。由于两个模型都被优化过，因此每个模型都会面对越来越强大的对手，这是单模型强化学习无法实现的。只要两个模型具有相当的实力且行为模式有所不同即可。在推理阶段，它们以训练时那样的方式协同工作，即一个模型生成解决方案，另一个模型在阅读该方案后给出回应。在DeepMath与Qwen3的艰难任务中，这种方式使得GRPO的pass@1得分提升了两倍，大约是未经训练的混合智能模型在相同基准上的得分的八倍。这种机制在竞赛编程任务以及不同模型家族中（如Qwen3.5、Gemma 4）也适用。目前，这些模型还是以文本形式进行交流；下一步就是让它们在潜在空间中共同进行推理。
English Abstract
Reinforcement learning from verifiable rewards (e.g. GRPO) is the engine behind today's reasoning models, yet it grades only the final answer. On hard problems this trains models to write more rather than to think better, since the trace itself is never graded and no label for good thinking exists. We introduce Agon, which makes two competing models each other's graders. Both attempt the same problem; in alternating roles, one drafts a solution and the other reads it while solving, and each is rewarded for out-solving the other. To win, a model must out-reason a rival that has seen its work, so reasoning is judged implicitly during training, with no process labels and no reward model. Because both models are optimized, each faces a progressively stronger rival, which single-model RL cannot provide. The two need only be comparably strong and behaviorally different. At inference the pair deploys as it trains, a two-stage cascade in which one model drafts and the other answers after reading the draft. On the hard split of DeepMath with Qwen3, this doubles GRPO's pass@1, roughly eight times the gain of an untrained Mixture-of-Agents pass over the same base. The ordering replicates on competitive-programming code and across model families (Qwen3.5, Gemma 4). For now the models talk in text; the next step is to let them reason together in latent space.