Reproduce “Motion Consistency Model: Accelerating Video Diffusion with Disentangled Motion-Appearance Distillation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yuanhao Zhai, Kevin Lin, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Chung-Ching Lin, David Doermann, Junsong Yuan and 1 more
Researchers from SUNY Buffalo and Microsoft developed the Motion Consistency Model (MCM) to accelerate video diffusion by achieving few-step sampling while simultaneously improving frame quality and style adaptability. MCM consistently outperformed existing distillation methods, showing lower FVD and higher CLIPSIM scores, and enabled zero-shot style transfer for generated videos in just 1-4 sampling steps.
Key insights
- Disentangling motion and appearance learning within a distillation framework is crucial to simultaneously achieve fast sampling and high visual quality in video generation, especially when leveraging image data.
- Utilizing high-quality *image* datasets, alongside video data, can effectively compensate for the limitations of low-quality *video* datasets to enhance the aesthetic fidelity and style of generated video frames.
- Mixing real and synthetically generated high-quality trajectories during distillation prevents mode collapse and improves the robustness and overall performance of the distilled video model.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.