Reproduce “Learning to Act with World Models by Valuing What Comes Next” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Chenghao Gao, Zhi Zhang, Dian Jin, Yihe Pan, Doudou Zhou

WM-PPO trains a policy using a world model to predict candidate actions’ successor observations and a critic to estimate their effect on success. Real branch outcomes calibrate these signals, while only the policy is used at inference. On ScienceWorld, it achieved 52.9% greedy success versus PPO-160’s 39.3%, averaged over three runs at comparable, not identical, real-interaction budgets.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.