Reproduce “Memory-V2V: Memory-Augmented Video-to-Video Diffusion for Consistent Multi-Turn Editing” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Dohun Lee, Chun-Hao Paul Huang, Xuelin Chen, Jong Chul Ye, Duygu Ceylan, Hyeonho Jeong
Memory-V2V introduces a memory-augmented framework for iterative video editing that maintains cross-turn consistency by integrating prior edits as structured constraints. It utilizes a hybrid retrieval and compression strategy, including dynamic tokenization and adaptive token merging, to achieve improved visual quality and consistency in novel view synthesis and text-guided long video editing, while significantly reducing computational cost.
Key insights
- Effective multi-turn video editing requires memory to function as a *constraint* mechanism on subsequent generations, rather than merely additional temporal context.
- A combination of selective, task-aware retrieval and dynamic resolution tokenization is crucial for scalably incorporating a large history of prior edits into generative models.
- Adaptive token merging based on a frame's 'responsiveness' allows for significant computational reduction without degrading visual quality or cross-turn consistency, by compactly representing less influential past frames.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.