Reproduce “WM-VLM: Probing Internal World Models for Interleaved Visual-Textual Reasoning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yuheng Zha, Yilei Wang, Qiyue Gao, Junrong Chen, Yujia Wu, Zhengfeng Lai, Zhengzhong Liu, Eric P. Xing
WM-VLM adds a visual-generation branch to a pretrained vision-language model to predict intermediate visual states and use them when answering 2D and 3D rotation questions. It is trained first on known states, then to reason from its predictions. On Tetris tests, it exceeds a fine-tuned text-token baseline by 16.30 to 39.25 percentage points; results do not establish transfer beyond synthetic rotations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.