Reproduce “Video-As-Prompt: Unified Semantic Control for Video Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yuxuan Bian, Xin Chen, Zenan Li, Tiancheng Zhi, Shen Sang, Linjie Luo, Qiang Xu
ByteDance's Intelligent Creation Lab and The Chinese University of Hong Kong developed Video-As-Prompt (VAP), a framework for unified semantic control in video generation. VAP employs reference videos as task-agnostic prompts within a Mixture-of-Transformers architecture, achieving a 38.7% user preference rate that rivals commercial models and demonstrating robust zero-shot generalization across diverse semantic conditions.
Key insights
- Unified semantic control can be achieved by reframing the problem as in-context generation, leveraging Diffusion Transformers with reference videos serving as abstract prompts.
- A Mixture-of-Transformers design is effective for preserving the generative capabilities of a frozen backbone while enabling a separate expert to learn and apply diverse semantic controls dynamically without catastrophic forgetting.
- Careful design of position embeddings, particularly a temporally biased Rotary Position Embedding, is critical to avoid imposing inappropriate pixel-level priors when transferring abstract semantic information between reference and target videos.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.