Reproduce “GPT4Motion: Scripting Physical Motions in Text-to-Video Generation via Blender-Oriented GPT Planning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jiaxi Lv, Yi Huang, Mingfu Yan, Jiancheng Huang, Jianzhuang Liu, Yifan Liu, Yafei Wen, Xiaoxin Chen and 1 more
GPT4Motion, a training-free framework for text-to-video generation, effectively produces videos with physically accurate motions. The approach leverages GPT-4's planning capabilities to script Blender physics simulations, which then guide a Stable Diffusion XL model for video synthesis, resulting in superior physical realism and motion consistency compared to baseline models.
Key insights
- Large Language Models can effectively act as planners to generate executable scripts for complex 3D physics simulations, simplifying the creation of realistic motion content.
- The integration of a physics engine (Blender) provides robust, physically accurate motion constraints (via edge and depth maps) that significantly improve the realism of generated videos.
- Combining dual ControlNet guidance with cross-frame attention in a diffusion model is crucial for maintaining both physical fidelity and temporal coherence, reducing flickering in synthesized video sequences.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.