Reproduce “StrucPhysVideo: Learning Physical Dynamics from Structured Captions and Robot Actions” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Awomo-WM Team, Enhui Ma, Kaiwen Guo, Tingrui Zhang, Wei Song, Yingshui Tan, Jianhua Xu, Tong Zhang and 1 more
StrucPhysVideo combines physics-focused video curation, structured temporal captions, and action-conditioned latent video generation to improve learned predictions of physical scene dynamics. Its text-image-to-video model reaches 45.5% on Physics-IQ Verified, while the robot-action model improves trajectory, depth, perceptual, and interaction metrics over DreamDojo-AgiBot.
Key insights
- Physics-focused captions improve physical-dynamics prediction beyond raw captions: Physics-IQ scores rise from 24.8% to 33.2% on Wan2.2-5B and from 37.9% to 45.5% on LingBot-Video-30B.
- Preserving the reference latent during sampling and explicitly representing temporal entity behavior helps maintain object identity, scene geometry, contacts, and plausible post-interaction motion.
- The framework treats physical world modeling as a combined data-quality, language-grounding, generative-modeling, and action-conditioning problem rather than relying solely on larger video models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.