问题出在哪儿？基于语义状态的Web代理过程级评估
Where Did It Go Wrong? Process-Level Evaluation of Web Agents with Semantic State Tracking
摘要
网络代理通过长时间的互动过程来执行任务。然而，现有的评估标准仅关注最终的成功情况，而忽略了所有与过程相关的信息，也无法提供有效的改进建议。在这项研究中，我们对网络代理进行了过程层面的分析。我们引入了WebStep这一评估工具，该工具包含了1,800个任务实例，这些任务的难度是可控的，同时还能自动跟踪系统的状态变化。每个应用程序都提供了一个确定的语义MDP模型，同时用户可以通过图形界面与系统交互。代理在界面上进行操作，而环境则会在后台记录系统的状态和转换过程，从而可以进行细致的分析，而无需手动标注数据。基于这些语义轨迹数据，我们发现，某些过程指标能够揭示那些在结果评估中难以观察到的差异：例如，三个代理的成功率都在31%-33%之间，但它们的探索能力和执行精度却有所不同。进一步按技能分类后，我们可以观察到不同技能之间的对比关系，比如，在Housing任务上，OpenAI CUA在提交动作方面的表现优于Qwen3.5，但在过滤任务上的表现则稍差了15.6%。这表明，即使在同一领域内，也有具体的技能需要改进。此外，我们还发现，导致任务失败的错误往往是由代理自身造成的，而不是共有的问题。最后，随着任务难度的增加，这些差异变得更加明显：在简单的任务上，成功率相当，但当任务难度增加时，成功率就会显著下降。我们的过程层面分析为网络代理的评估开辟了新的途径，为我们提供了关于如何改进每个代理的详细且实用的指导。
English Abstract
Web agents act through long interaction sequences, yet existing benchmarks evaluate only terminal success, discarding all process information and offering little guidance on improvement. In this work, we conduct a process-level analysis of web agents. We introduce WebStep, a benchmark of 1,800 task instances with controlled difficulty and automatic semantic state tracking. Each website exposes a deterministic semantic MDP alongside the GUI: the agent operates on the interface, while the environment records high-level states and transitions in the background, enabling fine-grained analysis without manual annotation. Based on the semantic trajectory, we first show that process metrics reveal differences invisible to outcome evaluation: three agents whose success rates cluster within 31-33% diverge in exploration reach versus execution accuracy. Then, decomposing by skill characterizes the nature of these differences, exposing opposite per-skill rankings hidden within the same website: e.g., on Housing, OpenAI CUA outperforms Qwen3.5 by 23.7% on commit actions yet underperforms it by 15.6% on filtering, pinpointing a concrete skill to improve even within a domain. Bifurcation analysis further localizes the decisive error that loses the task and shows that this error is agent-specific rather than shared. Finally, these differences widen as tasks grow harder: success rate is similar on easy tasks but separates sharply as exploration becomes more demanding. Our process-level analysis opens a new avenue in web agent evaluation, providing fine-grained and actionable insight into where and how each agent should be improved.