Reproduce “UnityShots: Memory-Driven Multi-Shot Audio-Video Generation with Boundary-Aware Gating” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jiehui Huang, Yuechen Zhang, Bin Xia, Jiahao Wang, Xu He, Zhenchao Tang, Meng Chu, Xin Tao and 2 more
UnityShots presents a memory-driven system for multi-shot audio-video generation, utilizing dual visual memory (long-term and short-term) and a persistent audio anchor combined with boundary-aware gating for dynamic memory updates. This system effectively mitigates identity and scene drift, producing coherent and consistent multi-shot sequences across various conditioning modes and cultural contexts.
Key insights
- Differentiating visual memory into long-term (anchoring opening shot elements) and short-term (capturing local motion) components is crucial for maintaining cross-shot coherence and minimizing drift.
- Dynamically modulating memory updates based on combined visual and audio boundary signals allows for more cinematically appropriate transitions and better content flow.
- A rule-based, non-suppressible contribution from the opening-shot anchor within the Long-Term Memory (LTM) is critical for preserving subject identity and narrative stability in longer sequences.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.