CODA-BENCH：代码代理能够处理那些需要大量数据处理的任务吗？
CODA-BENCH: Can Code Agents Handle Data-Intensive Tasks?
摘要
高级智能体在自主执行任务方面的潜力日益显现，因此对于能够反映现实世界开发场景复杂性的评估基准需求也越来越大。这类环境通常涉及到复杂的代码以及大规模的数据（例如文件系统）。然而，现有的评估基准往往只单独评估与代码或数据相关的能力，这显然无法完全反映真实开发场景的复杂性。在本文中，我们提出了CODA-BENCH这一基准测试工具，它是首个能够在数据密集型环境中同时评估代码处理能力和数据处理能力的基准测试工具。我们基于Kaggle生态系统构建了一个数据密集型Linux测试环境（包含数百个数据集），智能体需要在其中积极探索复杂的文件结构，以识别相关资源并生成用于数据分析的代码。CODA-BENCH包含了1,009个任务，这些任务涵盖了31种不同的应用场景；每个任务环境平均包含980个文件，从而模拟了真实的数据规模和噪声情况。对高级智能体的评估结果显示，即使是最优秀的系统也难以有效地将数据发现能力与代码执行能力结合起来，其成功率仅为61.1%。这些结果表明，当前智能体在数据处理方面的能力存在显著不足，同时也为未来的研究指明了方向。
English Abstract
Advanced agents are increasingly demonstrating the potential to operate as autonomous engineers, creating a growing demand for evaluation benchmarks that capture the complexity of real-world development. Such environments typically involve both complex code and large-scale data (i.e., file system). However, existing benchmarks usually evaluate code-centric or data-centric capabilities in isolation, leaving a clear gap with real development scenarios. In this paper, we bridge this gap by introducing CODA-BENCH, the first benchmark to jointly evaluate code and data intelligence in a data-intensive environment. We construct a data-intensive Linux sandbox based on the Kaggle ecosystem (containing hundreds of datasets), where agents must actively explore complex file hierarchies to identify relevant resources and generate code for data-driven analytical tasks. CODA-BENCH comprises 1,009 tasks spanning 31 communities, with each task environment containing an average of 980 files, simulating realistic data scale and noise. Evaluations of advanced agents reveal that even top-performing systems struggle to effectively integrate data discovery with code execution, achieving a success rate of only 61.1%. These results highlight a substantial gap in current agentic capabilities for data-intensive tasks and point to promising directions for future research.