Reproduce “GlobeBench: Evaluating Text-Based World Model Capabilities for Scalable Environment Simulations” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Nicholas Saban, Charvi Bannur, Mariya Vasileva, Darshan Deshpande

GlobeBench tests whether models predict environment transitions by comparing their answers with next states produced by executing actions in fixed software environments. Its items pair visible interaction histories with a final action and cover 19 skills. Across 20 models, success falls from 43.49% when nothing changes to 5.10% when unnamed entities change; most such labels are operator-assigned.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.