Reproduce “Learning Plug-and-play Memory for Guiding Video Diffusion Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Selena Song, Ziming Xu, Zijun Zhang, Kun Zhou, Jiaxian Guo, Lianhui Qin, Biwei Huang
Researchers from the University of California, San Diego and The University of Tokyo developed DiT-Mem, a plug-and-play memory module that guides video diffusion models by injecting external world knowledge. This method significantly enhances the physical consistency and rule adherence of generated videos, using frequency-aware disentanglement to improve realism without compromising visual quality.
Key insights
- DiT models effectively respond to in-context steering guidance, validating the approach of injecting external knowledge to influence generation.
- Frequency analysis within DiT's representations reveals that low-pass filtered features primarily capture object appearance, while high-pass filtered features encode motion and physical dynamics.
- The combination of frequency-disentangled memory streams through a shared self-attention module optimally balances enhancing physical motion with preserving visual fidelity to the text prompt.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.