PlanBench-XL：评估大型工具生态系统中LLM工具使用智能体在长期规划方面的表现
PlanBench-XL: Evaluating Long-Horizon Planning of LLM Tool-Use Agents in Large-Scale Tool Ecosystems
摘要
大型语言模型代理越来越多地运行在复杂的工具环境中，在这些环境中，实际任务需要找到合适的工具、推断隐含的子目标，并在长期范围内适应动态环境。然而，现有的评估标准很少考虑在工具可访问性受限情况下的规划问题。为了填补这一空白，我们提出了PlanBench-XL这个交互式评估平台，该平台包含327项零售任务和1,665种工具，用于测试代理是否能够反复获取可用的工具，利用这些工具来获取有助于实现最终目标的中间证据。PlanBench-XL还包含一种可选의 블록化机制，可以模拟现实中的不可预测情况，比如工具功能缺失、失效或干扰，从而迫使代理能够发现被中断的规划路径并做出调整。对十种主流大型语言模型的实验表明，在大量工具环境下进行规划仍然具有挑战性：虽然GPT-5.4在没有阻塞的情况下能达到51.90%的准确率，但在最严重的阻塞情况下，其准确率则下降到了11.36%。进一步分析显示，当工具故障没有明确的错误信号时，或者恢复过程需要更长的替代工具使用路径时，代理就会面临更大的困难。这些结果表明，PlanBench-XL是一个有效的测试平台，可用于诊断代理规划中的失败情况，同时也凸显了在具有大量不完美工具的长期任务中，需要强大的自适应规划能力。
English Abstract
LLM agents increasingly operate in large tool ecosystems, where real-world tasks require discovering relevant tools, inferring implicit sub-goals, and adapting to dynamic environments over long horizons. However, existing benchmarks rarely evaluate planning under retrieval-limited tool visibility. To address this gap, we introduce PlanBench-XL, an interactive benchmark of 327 retail tasks over 1,665 tools that tests whether agents can iteratively retrieve usable tools, invoke them to uncover intermediate evidence for subsequent calls toward the final goal. PlanBench-XL further features an optional blocking mechanism that simulates real-world unpredictability through missing, failing, or distracting tool functions, forcing agents to detect disrupted paths and adapt at runtime. Experiments on ten leading LLMs show that massive-tool planning remains challenging: while GPT-5.4 achieves 51.90% accuracy in block-free settings, it collapses to 11.36% under the most severe blocking condition. Further analysis shows that agents are especially vulnerable when failures lack explicit error signals or when recovery requires longer alternative tool-use paths. These results establish PlanBench-XL as a testbed for diagnosing agentic planning failures and highlight the need for robust adaptive planning in long-horizon tasks with large, imperfect tool environments.