Reproduce “The Flip Test: Video World Models Fail to Learn the World State” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Mi Luo, Lijun Yu, Adams Wei Yu, Ang Cao, Baochen Sun, Ming-Hsuan Yang, Nan Hua, Yuan Hao and 5 more

FlipTest evaluates whether video models change predictions across physical tipping points. It compares near-identical starting scenes, matched controls, and simulated sweeps to test whether predictions track the feature that changes the outcome. Across 11 tested models, Gemini Omni Flash 1.1 achieves 18.3% Level 1 pair accuracy, below the 25% binary-guessing reference; all tested video models score below 25% on simulation.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.