GameCraft-Bench：代理程序能否在真实的游戏引擎中从开始到结束完整地构建出可玩的游戏？
GameCraft-Bench: Can Agents Build Playable Games End-to-End in a Real Game Engine?
摘要
游戏生成是编码智能体技术的一种新兴应用方向，其核心在于将自然语言描述转化为可玩的互动系统。与传统编码任务不同，游戏生成是在游戏引擎内部进行的；在这个过程中，脚本、场景、资源、渲染效果以及运行时交互都需要协同工作，才能形成连贯的游戏体验。我们将游戏生成的整个过程定义为：通过目标环境中的玩家互动，实现从自然语言描述到完整游戏作品的过程。我们认为，评估这一过程需要满足三个要求：引擎的可靠性、作品的完整性，以及互动效果的验证性。我们提出了一个以互动效果为导向的评估框架，该框架通过重放演示和多元指标来评估游戏的可执行性。我们将该框架具体化为GameCraft-Bench测试集，该测试集包含140个基于Godot引擎开发的游戏案例，涵盖了15种不同的游戏类型。对前沿编码智能体的评估结果显示，完整的游戏生成仍然是一项极具挑战性的任务：最强的智能体也只能达到41.46%的成功率，而大多数智能体的得分都低于40%。进一步分析表明，虽然智能体能够实现一些可识别的游戏机制，但它们在创造具有丰富内容、功能完善的视觉反馈以及连贯呈现效果的完整游戏方面仍然存在困难。更多相关信息请访问https://tongxuluo.github.io/gamecraft-bench-website。
English Abstract
Game generation is an emerging application of coding agents, requiring models to transform natural-language specifications into playable interactive systems. Unlike traditional coding tasks, game generation takes place within a game engine, where scripts, scenes, assets, rendering, and runtime interactions must jointly produce coherent gameplay. We formalize end-to-end game generation as the problem of producing a complete game artifact that realizes a specification through observable player-game interaction in a target environment. We argue that evaluating this setting requires three desiderata: Engine Grounding, Artifact Completeness, and Interactive Verification. We propose an interaction-grounded evaluation framework that assesses executable gameplay through replayed demonstrations and rubric-guided multimodal judging. We instantiate this framework as GameCraft-Bench, a benchmark comprising 140 Godot tasks across 15 game families. Evaluations of frontier coding agents show that end-to-end game generation remains highly challenging: the strongest agent achieves only 41.46%, and most agents score below 40%. Further analysis reveals that while agents often implement recognizable mechanics, they struggle to deliver complete games with sufficient content, functional visual feedback, and coherent presentation. See https://tongxuluo.github.io/gamecraft-bench-website for demos, code, and data.