Reproduce “WM-VLM: Probing Internal World Models for Interleaved Visual-Textual Reasoning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yuheng Zha, Yilei Wang, Qiyue Gao, Junrong Chen, Yujia Wu, Zhengfeng Lai, Zhengzhong Liu, Eric P. Xing

WM-VLM adds a visual-generation branch to a pretrained vision-language model to predict intermediate visual states and use them when answering 2D and 3D rotation questions. It is trained first on known states, then to reason from its predictions. On Tetris tests, it exceeds a fine-tuned text-token baseline by 16.30 to 39.25 percentage points; results do not establish transfer beyond synthetic rotations.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.