Reproduce “MotionPhysics: Learnable Motion Distillation for Text-Guided Simulation” on OpenResearch

About this paper

Miaowei Wang, Jakub Zadrożny, Oisin Mac Aodha, Amir Vaxman

MotionPhysics introduces a framework for text-guided 3D dynamic simulation, integrating multimodal LLMs for parameter inference with a differentiable Material Point Method simulator. The system employs a novel Learnable Motion Distillation loss to extract pure motion signals from video diffusion models, enabling realistic and physically plausible animations for diverse materials and challenging out-of-distribution objects.

Key insights

• Pre-trained video diffusion models can be effectively utilized as motion priors for physical simulation if their inherent appearance and geometric biases are actively suppressed during distillation.
• Grounding LLM-based physical parameter initialization with real-world material property bounds through 'constraint-aware prompts' is crucial for preventing hallucinations and ensuring simulation stability.
• A fully differentiable pipeline, combining LLM-driven initialization, advanced physical simulation, and an appearance-decoupled motion loss, enables generalizable text-guided dynamic content creation.

Summary by alphaXiv.