Reproduce “GameASG-Bench: Benchmarking Autonomous Software Generation for Game Development” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xiuhui Zhang, Yi Chen, Shusheng Xu, Fan Li, Huan Wang, Tongkai Yang, Binhang Yuan
GameASG-Bench tests whether generated browser games satisfy required behavior, not merely launch. Before generation, task authors define an interface for scenarios, player actions, state snapshots, and checks using source inspection and Chromium. Across 47 tasks, GPT-6-Astra with Codex CLI reached a 93.2% mean L2 pass rate but strict success on 26 tasks (55.3%). Results are limited to the benchmark.
Key insights
- High average check scores do not guarantee complete task compliance: the best configuration reached a 93.2% mean L2 pass rate but achieved strict success on only 26 of 47 tasks.
- Behavioral integration is the main bottleneck; agents may implement expected source structures while failing to connect evaluator actions, internal state, visible rendering, and gameplay progression.
- More resources help unevenly: full tool access and larger turn budgets improve completion, while reasoning effort shows a non-monotonic trade-off between check-level performance, strict success, and token cost.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.