Reproduce “COVE: Unleashing the Diffusion Feature Correspondence for Consistent Video Editing” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jiangshan Wang, Yue Ma, Jiayi Guo, Yicheng Xiao, Gao Huang, Xiu Li
Leveraging inherent diffusion features, COVE introduces a framework for consistent video editing that adaptively tracks token correspondences across frames. The method produces high-quality video edits with enhanced temporal coherence, outperforming existing baselines in various metrics and user preference without requiring additional model training.
Key insights
- Diffusion features inherently contain rich, fine-grained, and robust "one-to-many" correspondence information that can be effectively leveraged for temporal consistency in video.
- Explicitly guiding attention with these intrinsic correspondences is crucial for achieving fine-grained temporal consistency, surpassing reliance on external optical flow or global attention.
- A localized, sliding-window search strategy for correspondence acquisition significantly improves computational efficiency, making the method practical for video editing.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.