训练后被忽视的免费午餐：对LLM智能体来说是一种优势
Neglected Free Lunch from Post-training: Progress Advantage for LLM Agents
摘要
过程奖励模型能够实现对LLM进行细粒度、步骤级别的评估，但针对代理式场景构建这样的模型却非常困难：长周期的交互、不可逆转的操作以及随机性的环境反馈使得人工注释和蒙特卡洛估计在大规模应用中难以实现。在本研究中，我们发现强化学习在训练后就能提供有效的步骤级评分机制，从而完全无需专门训练奖励模型。具体来说，我们在一般随机马尔可夫决策过程中推导出了一种隐式优势函数，我们将其称为“进展优势”——即强化学习训练得到的策略与其参考策略之间的对数概率比，恰好能够复现最优优势函数。这种形式使得所得到的信号无需进行注释处理，且不受领域限制，同时还可以作为标准强化学习训练流程的副产品得到利用。我们在五个基准测试和四种模型家族上验证了这种进展优势的有效性：在测试时扩展、不确定性量化以及故障归因等方面，它都优于基于置信度的基线方法；而且尽管不需要针对特定任务进行专门训练，它仍然优于那些需要专门训练的奖励模型。我们通过对进展优势特性的更深入分析，为其在实际代理系统中应用提供了实用指导。
English Abstract
Process reward models enable fine-grained, step-level evaluation of LLMs, yet building them for agentic settings remains prohibitively difficult: long-horizon interactions, irreversible actions, and stochastic environment feedback make both human annotation and Monte Carlo estimation infeasible at scale. In this work, we show that reinforcement learning (RL) post-training already provides the ingredients for effective step-level scoring, eliminating the need for dedicated reward model training altogether. Concretely, we derive an implicit advantage under a general stochastic Markov decision process, which we term progress advantage -- log-probability ratio between the RL-trained policy and its reference policy exactly recovers the optimal advantage function. This formulation makes the resulting signal annotation-free, domain-agnostic, and available as a byproduct of the standard RL post-training pipeline. We validate the effectiveness of the progress advantage across three different applications: test-time scaling, uncertainty quantification, and failure attribution on five benchmarks and four model families. Across all settings, it consistently outperforms confidence-based baselines and, despite requiring no task-specific training, surpasses dedicated trained reward models. We complement these results with deeper analyses on characteristics of progress advantage, offering practical guidance for adoption in real-world agentic systems.