Reproduce “World Models in Words: Auditing Physical State-Transition Commitments in Vision-Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Emmanuelle Bourigault
Researchers from the University of Oxford introduce WORLD MODELS IN WORDS, an evaluation framework for auditing the language-expressed physical state-transition commitments of Vision-Language Models (VLMs). The framework reveals that 18% to 42% of VLMs' correct answers are supported by physically invalid reasoning and demonstrates that preference tuning can reduce these inconsistencies while improving accuracy.
Key insights
- VLMs frequently provide correct answers despite exhibiting physically inconsistent or invalid reasoning traces, indicating a substantial gap between superficial performance and robust physical understanding.
- For most VLMs, predicting correct physical transitions is a greater challenge than visually extracting the initial state, suggesting a bottleneck in dynamic reasoning rather than solely perception.
- Verifier-guided reranking of sampled traces and DPO-style preference training can effectively improve VLM physical consistency and reduce hidden inconsistencies without negatively impacting final-answer accuracy.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.