为什么多步骤工具使用强化学习机制会崩溃，以及监督信号如何解决这一问题
Why Multi-Step Tool-Use Reinforcement Learning Collapses and How Supervisory Signals Fix It
摘要
工具使用使得大型语言模型能够完成复杂的任务。最近的代理式强化学习方法在提升模型能力方面具有巨大潜力。然而，仅依靠强化学习往往会导致模型不稳定或性能受限。在我们的实验中，一些模型会出现灾难性崩溃现象，即模型性能突然下降，工具调用机制也失效。分析表明，这种故障源于某些控制标记上出现的异常概率变化，从而破坏了模型的正常运作流程，但模型本身的工具使用能力仍然完好无损，只是被特定的格式所掩盖了。为了解决这一问题，我们系统地研究了多种监督信号，包括非策略性监督、基于提示的引导、错误样本监督等，并分别在同步训练和交错训练模式下应用这些信号。我们发现，将监督微调与强化学习相结合可以显著提高模型的稳定性，但在超出分布范围的格式和内容评估中，模型性能会下降。我们还分析了学习率以及不同场景下的泛化能力。这些结果强调了理解强化学习中的故障的重要性，同时也表明多种监督信号可以帮助模型进行探索性学习，从而帮助大型语言模型能够稳定地完成复杂的多步骤工具使用任务。我们的代码可以在https://github.com/hypasd-art/Tool-RL-Box中找到。
English Abstract
Tool use enables large language models (LLMs) to perform complex tasks, and recent agentic reinforcement learning (RL) methods show promise for enhancing model capabilities. However, RL alone often leads to instability or limited gains in tool-use tasks. In our experiments, some models exhibit catastrophic collapse, where performance abruptly drops and tool-invocation structures fail. The analysis reveals that these failures stem from unexpected probability spikes in specific control tokens, disrupting structured execution, yet the underlying tool-use capability remains intact, merely obscured by specific formats. To address this, we systematically investigate a diverse set of supervisory signals, including off-policy supervision, hint-based guidance, erroneous example supervision, and others, applied under both synchronous and interleaved training schemes. We find that interleaving supervised fine-tuning (SFT) with RL substantially improves stability, but exhibits degraded performance under format and content out-of-distribution (OOD) evaluation. We also analyze the impact of learning rates and generalization across settings. These results highlight the importance of understanding RL failures and demonstrate how diverse supervisory signals can guide exploratory learning, enabling robust training of LLMs for complex, multi-step tool-use tasks. Our Code is available at https://github.com/hypasd-art/Tool-RL-Box.