Reproduce “The Flip Test: Video World Models Fail to Learn the World State” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Mi Luo, Lijun Yu, Adams Wei Yu, Ang Cao, Baochen Sun, Ming-Hsuan Yang, Nan Hua, Yuan Hao and 5 more
FlipTest evaluates whether video models change predictions across physical tipping points. It compares near-identical starting scenes, matched controls, and simulated sweeps to test whether predictions track the feature that changes the outcome. Across 11 tested models, Gemini Omni Flash 1.1 achieves 18.3% Level 1 pair accuracy, below the 25% binary-guessing reference; all tested video models score below 25% on simulation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.