Reproduce “Memorize-and-Generate: Towards Long-Term Consistency in Real-Time Video Generation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Tianrui Zhu, Shiyi Zhang, Zhirui Sun, Jingqi Tian, Yansong Tang

The Memorize-and-Generate (MAG) framework from Tsinghua University addresses long-term historical consistency in real-time video generation by integrating a learnable memory compression model with a specialized generator. This approach achieves near-lossless memory compression and maintains scene integrity over extended sequences, delivering an inference speed of 21.7 FPS and outperforming previous autoregressive methods on historical consistency benchmarks like MAG-Bench.

Key insights

  • Decoupling historical memory compression from frame generation allows for specialized optimization, enabling both memory efficiency and high fidelity in retaining past scene information.
  • Forcing autoregressive models to generate frames with an empty text condition during training significantly improves their ability to infer and preserve physical and temporal consistency from historical context alone.
  • A learnable encode-decode mechanism for KV cache compression can achieve near-lossless historical information retention while drastically reducing memory footprint and maintaining real-time inference speeds.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.