Reproduce “JAMER: Project-Level Code Framework Dataset and Benchmark on Professional Game Engines” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jianwen Sun, Chuanhao Li, Zizhen Li, Yukang Feng, Fanrui Zhang, Yifei Huang, Yu Dai, Kaipeng Zhang
This work introduces JamSet, a project-level game code framework dataset, and JamBench, a corresponding benchmark, for the Godot engine, alongside a deterministic, multi-level verification pipeline. It reveals current AI models struggle with architectural design and cross-file coherence in game development tasks at scale, while demonstrating that fine-tuning on JamSet can guide models towards adopting human-like engineering practices.
Key insights
- Current large language models exhibit a 'capability cliff' in game code generation, with performance significantly degrading as project scale and complexity increase.
- Code Agents primarily improve syntactic correctness and compilability but show limited impact on architectural design, cross-file logical coherence, or high-level runtime behavioral quality.
- Model-generated projects often fail to adopt established game engineering best practices, such as using abstract input actions or autoload scripts, indicating an 'engineering paradigm gap'.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.