Reproduce “Enhancing Video Physical Consistency via Role-aware Joint Training and Modality-decoupled Denoising” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Guangting Zheng, Haojing Chen, Hao Li, Jingtao Zhang, Zhen Yang, Xiaosong Jia, Xue Yang, Shaofeng Zhang and 1 more
The Video Physical-consistency Tuning (VPT) framework enhances the physical consistency of generated videos by integrating role-aware dynamics and a modality-decoupled denoising strategy. It achieved up to a 39.4% relative gain in Semantic Adherence and an 17.9% relative gain in Physical Commonsense on the VideoPhy benchmark, while also improving overall video quality scores on VBench.
Key insights
- Distinguishing between different entity roles in a scene through a role-aware map provides crucial high-level physical structure for more realistic motion generation.
- Decoupling the denoising process for visual and auxiliary modalities, along with annealing auxiliary loss weights, is critical to prevent "capacity conflicts" and preserve the model's core visual generation capabilities.
- Leveraging physical direction learned by a final model relative to an intermediate one via cross-step auto-guidance offers a robust inference mechanism that enhances physical dynamics without being vulnerable to error accumulation from self-predicted auxiliary data.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.