NatureBench：编码智能体能否达到Nature系列论文中已发表的最先进水平？
NatureBench: Can Coding Agents Match the Published SOTA of Nature-Family Papers?
摘要
我们推出了NatureBench这一跨领域基准测试工具。该工具包含90项任务，这些任务均来自经过同行评审的《Nature》系列期刊论文。其目的就是评估人工智能编程智能体是否能够超越简单的复制行为，真正在真实的科学问题中实现创新。NatureBench基于NatureGym构建而成——这是一个自动化处理流程，能够从原始论文中生成标准化的、针对每项任务的专用环境。这种处理方式能够解决以往那些因环境复杂性而降低评估可信度的基准测试问题。在严格的网络搜索限制条件下对10种前沿智能体配置进行了评估，结果显示，在g>0.1的标准下，只有17.8%的任务中最强的模型能够超越当前最先进的模型。分析发现，这些智能体成功的关键在于它们将科学任务转化为熟悉的监督预测问题，而非真正的科学创新。失败则主要源于错误的方法选择以及计算资源不足，而非对任务理解的错误。我们同时发布了这个基准测试工具、NatureGym处理流程，以及包含维护方重复结果的公开排行榜。代码链接：https://github.com/FrontisAI/NatureBench
English Abstract
We introduce NatureBench, a cross-discipline benchmark of 90 tasks distilled from peer-reviewed Nature-family publications, designed to evaluate whether AI coding agents can move beyond reproduction toward discovery on real scientific problems. NatureBench is built on NatureGym, an automated pipeline that constructs a standardized, per-task containerized environment from a source paper, addressing the environment-fragmentation problem that has limited the credibility of prior agent-on-research benchmarks. Evaluating ten frontier agent configurations under a strict web-search-disabled protocol, we find that the strongest model surpasses SOTA on only 17.8% of tasks under the g>0.1 criterion. Analysis of method pathways reveals that agents succeed primarily through methodological translation, converting scientific tasks into familiar supervised prediction problems, rather than through genuine scientific invention. Failures are dominated by wrong method choice and insufficient compute budget, not by task misunderstanding. We release the benchmark, the NatureGym pipeline, and a public leaderboard with maintainer-side reproduction. Code: https://github.com/FrontisAI/NatureBench