Reproduce “IntPhys 2: Benchmarking Intuitive Physics Understanding In Complex Synthetic Environments” on OpenResearch

About this paper

Florian Bordes, Quentin Garrido, Justine T Kao, Adina Williams, Michael Rabbat, Emmanuel Dupoux

Researchers at Meta's FAIR laboratory developed IntPhys 2, a benchmark to evaluate intuitive physics understanding in AI systems. The benchmark features photorealistic environments, complex occlusion scenarios, and a "violation of expectation" framework, revealing that state-of-the-art AI models achieve near-chance performance (52-58% accuracy) while humans achieve 96% accuracy.

Key insights

• A substantial gap exists between human and state-of-the-art AI performance in intuitive physics, with AI models performing near chance levels (50-58%) on IntPhys 2 compared to human accuracy (96%).
• Current AI architectures, including MLLMs and predictive models, struggle with maintaining object representations during extended occlusions and with temporal reasoning in visually complex environments.
• The significant performance drop from previous benchmarks suggests that current models may rely on superficial visual patterns rather than developing a genuine understanding of physical principles.

Summary by alphaXiv.