Reproduce “In-Context Learning with Unpaired Clips for Instruction-based Video Editing” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xinyao Liao, Xianfang Zeng, Ziye Song, Zhoujie Fu, Gang Yu, Guosheng Lin
Researchers at Nanyang Technological University and StepFun developed a data-efficient, two-stage training strategy for instruction-based video editing that leverages unpaired video clips for pretraining. Their model achieved a 12% improvement in editing instruction following and a 15% improvement in editing quality compared to existing methods, while also enhancing video generation consistency and smoothness.
Key insights
- Utilizing abundant unpaired video clips for pretraining effectively establishes foundational editing capabilities and the ability to preserve original video content without expensive paired data.
- Training solely on large synthetic editing datasets for pretraining introduces artifacts and yields poorer performance compared to pretraining with real-world video clips.
- The in-context video injection mechanism, by assigning a zero timestep to original video tokens, enables the model to distinguish and leverage noise-free source content for high-quality, instruction-guided edits.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.