JAMER：专业游戏引擎层面的代码框架数据集与评估标准
JAMER: Project-Level Code Framework Dataset and Benchmark on Professional Game Engines
摘要
当前由人工智能驱动的游戏开发在资源生成、游戏玩法设计以及基于网络的游戏代码编写方面取得了显著进展。不过，由于缺乏大规模数据集和可靠的评估方法，专业游戏引擎上的项目级代码工程仍然处于未开发的阶段。我们提出了JamSet和JamBench——这是首个基于专业游戏引擎构建的项目级游戏代码框架数据集与评估工具。我们的核心发现是：游戏挑战赛这类社区活动让开发者在有限时间内完成完整的游戏开发，从而产生了数千个适合此用途的开源项目。利用Godot引擎的文本格式和无服务器运行模式，我们设计了一种从文件完整性到运行时行为评估的确定性验证流程，最终从24万多个代码仓库中筛选出8,133个经过验证的项目。其中，300个项目被纳入JamBench，其余则构成JamSet。JamBench通过主题驱动的生成和代码补全任务进行评估，评估标准包括编译通过率、结构完整性评分和行为一致性评分。对9种前沿模型的评估表明，随着项目规模增大，运行时通过率从小项目的80.4%下降到大项目的5.7%（Task2a）。虽然代码代理提升了编译效率，但运行时行为质量并未改善，说明瓶颈在于架构设计而非语法正确性。实验证明JamSet作为有效训练数据非常有效。所有数据和代码均可公开获取。
English Abstract
Current AI-driven game development has made substantial progress in asset generation, gameplay design, and web-based game coding, yet project-level code engineering on professional game engines remains largely unexplored due to the absence of large-scale datasets and deterministic evaluation methods. We present JamSet and JamBench, the first project-level game code framework dataset and benchmark built on a professional game engine. Our key insight is that Game Jam competitions, community events where developers build complete games under tight time constraints, yield thousands of open-source projects suitable for this purpose. Building on the Godot engine's text-based format and headless execution mode, we design a deterministic verification pipeline from file integrity to runtime behavior collection, distilling 8,133 verified projects from over 240,000 repositories. Of these, 300 manually verified projects form JamBench; the rest constitute JamSet. JamBench defines theme-driven generation and code completion tasks, evaluated through a pipeline combining compilation pass rates, Structural Completeness Score (SCS), and Behavioral Alignment Score (BAS). Evaluation of 9 frontier models reveals a capability cliff as project scale increases, with runtime pass rates dropping from 80.4% on small projects to 5.7% on large ones (Task2a). Code Agents improve compilation rates yet yield no gains in runtime behavioral quality, indicating that the bottleneck lies in architectural design rather than syntactic correctness. Experiments validate JamSet as effective training data. All data and code are publicly available.