Reproduce “Order from Chaos: Physical World Understanding from Glitchy Gameplay Videos” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Meng Cao, Haoran Tang, Haoze Zhao, Mingfei Han, Ruyang Liu, Qiang Sun, Xiaojun Chang, Ian Reid and 1 more
Researchers from Mohamed bin Zayed University of Artificial Intelligence and Peking University introduced a novel training paradigm that leverages glitches in gameplay videos to enhance Multi-modal Large Language Models' (MLLMs) physical world understanding. Their PhysGame dataset and GameBench benchmark led to performance improvements across various models, including a 9.4% absolute gain for InternVL2.5-8B on GameBench, and consistent positive transfer to real-world physical reasoning and general video understanding benchmarks.
Key insights
- Explicitly modeling violations of physical rules (glitches) in gameplay videos can lead to a deeper understanding of underlying physical principles.
- Incorporating gameplay meta-information (e.g., video titles, descriptions) into the prompting process significantly improves the accuracy and quality of automatically generated instruction-tuning data.
- Physical knowledge acquired from glitchy gameplay videos demonstrates strong transferability, enhancing MLLMs' performance on both real-world physical reasoning tasks and broader general video understanding benchmarks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.