Reproduce “World Models Meet Language Models: On the Complementarity of Concrete and Abstract Reasoning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yucheng Zhou, Wei Tao, Yiwen Guo, Jianbing Shen

This research introduces a framework to enable multimodal large language models (MLLMs) to intelligently control and integrate concrete visual simulations from world models into abstract reasoning for future-oriented visual prediction tasks. The proposed method, Privileged-Future On-Policy Self-Distillation (PF-OPSD), achieved an accuracy of 72.3% on VRQABench and 70.4% on OpenWorldQA, improving over supervised fine-tuning baselines by more than 10 percentage points.

Key insights

  • Effective integration of world models requires MLLMs to learn how to control their use and critically evaluate their outputs, rather than just having access to them.
  • Privileged information (ground-truth future videos and answers) during training can be leveraged to effectively teach an MLLM a self-calibrating policy for interacting with fallible generative tools.
  • Calibrated supervision of intermediate concrete-reasoning actions (e.g., when to simulate, how to verify, how much to rely) is crucial for performance gains, going beyond simple end-to-end imitation learning.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.