运行还是不运行：分析基于LLM的程序修复中代码执行的成本效益
To Run or Not to Run: Analyzing the Cost-Effectiveness of Code Execution in LLM-Based Program Repair
摘要
基于LLM的程序修复代理通常采用“生成-运行-修改”的模式来执行修复操作，通过反复测试来评估和优化修复方案。这种基于执行的策略已成为先进系统中的标准做法。不过，这种执行过程可能耗时且成本较高，而其对这些代理的影响却尚未得到充分研究。在本文中，我们对基于LLM的程序修复中的执行行为进行了两阶段实证研究。为了能够全面了解执行行为，我们首先分析了SWE-bench排行榜上7,745个代理的执行记录。其次，我们在200个SWE-bench实例中，对3,000次端到端的修复尝试进行了评估，这些尝试分别使用了三种代理——Claude Code、Codex和开源的OpenCode——并且采用了四种不同的执行模式，从而可以细致地比较性能和成本。我们的分析揭示了三个关键发现：(1) 所有被分析的代理和模型都采用了代码执行方式，平均每个任务需要8.8次测试运行。不同代理和模型的执行行为差异很大，频率从每个任务2次到19次不等，而后期阶段的执行通常具有更高的成功率。(2) 执行限制对修复成功率几乎没有影响：对于使用先进模型的商业代理来说，“禁止执行”与“无限制执行”之间的解决率差距仅为1.25个百分点，不具备统计学显著性；而“禁止执行”则能大幅减少标记数和时间成本。(3) 执行带来的好处并非均匀分布。这些现象表明，当前代理只是盲目地执行操作，而在那些无法带来明显好处的任务上则浪费了执行资源。因此，执行应被视为一种需要权衡成本和收益的资源，而不是默认具备的能力。
English Abstract
LLM-based agents for program repair are increasingly built on a "generate-run-revise" paradigm, iteratively executing tests to evaluate and refine patches. This execution-based approach has become standard practice in state-of-the-art systems. However, executions can be time-consuming and expensive, yet their impact on these agents remains underexplored. In this paper, we conduct a two-stage empirical study over execution behavior in LLM-based program repair. To characterize execution behavior at scale, we first analyze 7,745 agent traces from SWE-bench leaderboard submissions. Second, we evaluate 3,000 end-to-end repair attempts across 200 SWE-bench instances and three agents (Claude Code, Codex, and the open-source OpenCode) under four execution paradigms, which allows for a fine-grained comparison of performance and cost. Our analysis reveals three key observations: (1) Code execution is used across all agents and models analyzed, with an average of 8.8 test runs per task. Execution behavior varies substantially across agents and models, with frequency ranging from 2 to 19 per task, and late-stage executions consistently achieve higher success rates than early-stage ones. (2) Execution restrictions have little effect on repair success: on commercial agents with SOTA models the resolve-rate gap between Prohibited and Unrestricted is only 1.25 percentage points and not statistically significant, while Prohibited saves substantial token and wall-clock cost. (3) Execution benefit is concentrated rather than uniform. These patterns suggest that current agents apply execution indiscriminately, paying its cost on instances where it provides little benefit. Execution, therefore, should be treated as a resource with an explicit cost-benefit tradeoff, not a default capability.