Reproduce “Principia: Relational Physics Tests for Video Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Varun Varma Thozhiyoor, Shivam Tripathi, Venkatesh Babu Radhakrishnan, Anand Bhattad
A new benchmark called Principia employs calibration-independent relational physics tests to evaluate video generation models and vision-language models, revealing that while current models achieve high visual fidelity, their physical consistency scores range from only 0.14 to 0.42, and VLMs struggle to detect these physical violations, performing near chance.
Key insights
- Physical reasoning can be rigorously evaluated by focusing on calibration-independent relational invariants between objects, detectable directly in image space.
- There is a significant decoupling between visual quality and physical fidelity in state-of-the-art video generation models; visual realism does not imply physical understanding.
- Merely scaling up model size in both video generators and Vision-Language Models does not uniformly improve physical consistency or the ability to detect violations, and can even lead to regressions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.