Reproduce “RPGBENCH: Evaluating Large Language Models as Role-Playing Game Engines” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Pengfei Yu, Dongming Shen, Silin Meng, Jaewon Lee, Weisu Yin, Andrea Yaoyun Cui, Zhenlin Xu, Yi Zhu and 3 more
RPGBENCH, a benchmark from Boson AI and the University of Illinois Urbana-Champaign, evaluates Large Language Models (LLMs) as comprehensive text-based role-playing game engines, assessing their ability to both create game worlds and simulate interactive gameplay. It demonstrates that while LLMs excel at generating engaging narratives, they consistently struggle with maintaining mechanical consistency and rule adherence in complex game environments.
Key insights
- LLMs exhibit strong capabilities in generating creative and engaging narratives and adhering to complex JSON formatting for game creation.
- However, LLMs consistently struggle with the logical rigor and mechanical consistency required for enforcing game rules and updating state variables accurately during both game creation and simulation.
- There is a clear trade-off between mechanical consistency and narrative interestingness; lower sampling temperatures improve rule adherence but can reduce creative aspects.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.