Reproduce “OneStory: Coherent Multi-Shot Video Generation with Adaptive Memory” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhaochong An, Menglin Jia, Haonan Qiu, Zijian Zhou, Xiaoke Huang, Zhiheng Liu, Weiming Ren, Kumara Kahatapitiya and 7 more
Researchers from Meta AI and the University of Copenhagen developed OneStory, a framework for coherent multi-shot video generation that utilizes adaptive memory modules to model long-range cross-shot context. The method consistently outperforms existing baselines, achieving higher inter-shot coherence scores (e.g., 0.5813 average inter-shot coherence in text-to-multi-shot video tasks) and enhanced shot-level quality.
Key insights
- Reformulating multi-shot video generation as a next-shot autoregressive task, conditioned on prior shots, effectively leverages existing strong I2V models and allows for scalable, progressive narrative construction.
- Adaptive memory mechanisms that intelligently select and compress semantically relevant frames from historical context are critical for maintaining long-range narrative coherence across discontinuous shots, outperforming static or limited context approaches.
- A high-quality multi-shot video dataset with referential, shot-level captions is crucial for training models to understand and generate complex, evolving narratives that reflect realistic storytelling patterns.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.