RODS：以奖励为驱动的在线数据整合技术，用于多轮工具使用型智能体
RODS: Reward-Driven Online Data Synthesis for Multi-Turn Tool-Use Agents
摘要
多轮工具使用的强化学习算法面临着静态数据集中信息性样本迅速减少的问题。我们发现，GRPO中的梯度信号主要集中在那些具有最大奖励波动性的任务上，这是Popoviciu上界所带来的结果。因此，位于智能体能力边界附近的样本——即成功与失败大致平衡的区域——会产生巨大的策略梯度。随着训练的进行，这一边界会不断移动，从而逐渐消耗掉静态数据集中信息性样本的数量。我们提出了RODS（基于奖励的在线数据合成）方法来解决这一问题。RODS通过将训练过程中的奖励波动作为一种实用的、零成本的边界检测机制来连接强化学习训练与数据生成过程，无需额外的推理步骤，只需使用已经计算出的训练数据即可。它能够持续识别这些边界样本，并通过与技能相关的采样流程来生成符合其结构复杂度的新多轮样本（例如API拓扑和依赖关系深度），同时还能管理一个与策略共同进化的动态重放缓冲区。从400个人工种子开始，通过维持约800个活跃训练样本，RODS能够取得与17K样本离线算法相当的性能，而所需的数据量则少大约20倍。在我们的控制环境中，RODS的表现也优于固定数据驱动的强化学习和环境增强方法。
English Abstract
Multi-turn tool-use RL is bottlenecked by the rapid depletion of informative samples in static datasets. We observe that the gradient signal in GRPO concentrates on tasks with the highest rollout reward variance, a consequence of the Popoviciu upper bound. Consequently, samples near the agent's capability boundary -- where successes and failures are roughly balanced -- contribute disproportionately large policy gradients. As training progresses, this boundary continuously shifts, which gradually depletes the pool of informative samples in a static dataset. We propose RODS (Reward-driven Online Data Synthesis) to resolve this depletion. RODS closes the loop between RL training and data generation by repurposing the progress reward variance as a practical, zero-cost boundary detector that requires no extra inference beyond the rollouts already computed for training. It continuously identifies such boundary samples, synthesizes new multi-turn variants matching their structural complexity (e.g., API topology and dependency depth) via a skill-aligned resampling pipeline, and manages a dynamic replay buffer that co-evolves with the policy. Starting from 400 human seeds and maintaining an active training pool of ~800 samples, RODS achieves comparable performance to a 17K-sample offline pipeline while requiring roughly 20x fewer trajectories, and improves over fixed-data RL and environment augmentation in our controlled setting.