Reproduce “VIVA: VLM-Guided Instruction-Based Video Editing with Reward Optimization” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xiaoyan Cong, Haotian Yang, Angtian Wang, Yizhi Wang, Yiding Yang, Canyu Zhang, Chongyang Ma
Researchers from Brown University and ByteDance developed VIVA, a framework combining a VLM instructor and reward optimization for instruction-based video editing. VIVA performs complex edits on diverse real-world instructions, outperforming open-source baselines and achieving results comparable to commercial models in instruction following, source preservation, and editing quality.
Key insights
- Integrating a VLM instructor provides semantically rich, visually-grounded context, effectively disambiguating complex instructions for more precise video editing.
- Optimizing directly for instruction following, source video preservation, and human preferences through a reward-based reinforcement learning approach (Edit-GRPO) leads to higher quality and more faithful edits.
- Leveraging a synthetic data generation pipeline, augmented with image editing data, is crucial for training robust video editing models and enhancing their generalization to diverse and complex scenarios.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.