Reproduce “MotionBlind: Probing the Illusion of Motion Understanding in Video-LLMs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Dhairya Bhatia, Bishoy Galoaa, Oliver Fritsche, Shahid Kamal, Muhammad Obaidullah Abdul Salam, Umer Saleem, Om Rastogi, Frania Felix Chettiar and 2 more
MotionBlind tests whether video-language models read physical motion rather than scene appearance or language cues. It pairs near-identical clips with complementary questions and scores all four answers jointly, giving a 6.25% chance floor. On 60 self-recorded instances, Eagle2.5-8B reaches 11.7% IAcc, while Gemini 3.1 Pro reaches 60.0% overall but 14.3% on speed within this controlled setting.
Key insights
- Open-source Video-LLMs exhibit an 'illusion of motion understanding,' achieving near-chance Instance Accuracy (IAcc) despite higher per-item accuracy, indicating an inability to consistently discern subtle motion differences in contrastive scenarios.
- MotionBlind genuinely requires both visual input and correct temporal ordering, as integrity probes (text-only, shuffled/reversed frames) collapsed IAcc to 0% for all models, validating its robustness against shortcuts prevalent in other benchmarks.
- Models demonstrate specific 'blindness' to speed and magnitude categories, with IAcc remaining at or near 0% for open models and only 14% even for Gemini 3.1 Pro on 'speed,' suggesting a fundamental representational gap for kinematic properties.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.