Reproduce “Inferring Dynamic Physical Properties from Video Foundation Models” on OpenResearch

About this paper

Guanqi Zhan, Xianzheng Ma, Weidi Xie, Andrew Zisserman

The authors introduce PhysVid, a benchmark and method for estimating elasticity, viscosity, and dynamic friction from video. The method freezes pretrained video-model representations and trains a learned readout to predict properties; the study evaluates prompted multimodal language models. On real relative-friction comparisons, V-JEPA-2 improves from 0.48 to 0.87 ROC AUC after real-data adaptation, matching the oracle, but transfer remains property-dependent.

Key insights

• Video generative (DynamiCrafter) and self-supervised (V-JEPA-2) models can learn useful representations for inferring dynamic physical properties, achieving strong performance on synthetic data.
• Current video foundation models exhibit a noticeable performance gap compared to physics-informed oracle methods, especially for absolute value prediction and generalizing to real-world scenarios without specific domain adaptation.
• MLLMs generally perform worse than dedicated video models on quantitative physical tasks but can benefit from tailored prompting strategies, sometimes relying more on semantic cues than direct kinematic motion, particularly in real-world contexts.

Summary by alphaXiv.