Reproduce “From Game Generation to Game Playing: Fully Automated Ability Training for Vision-Language-Action Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Changran Liu, Qi Wu, Wenchao Ma, Sharon X. Huang, Haomiao Jiang

SCRIPT2PLAY trains a 4B vision-language model to play from images and instructions by cloning scripted experts in generated games. Experts use hidden engine state to plan but record keyboard-and-mouse actions; only verified successful rollouts train the model. With 150.3 hours of synthetic demonstrations, it reaches 69.6% average OpenHA success and a 0.640 mean on five OmniGameArena games, showing transfer in evaluated settings.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.