Reproduce “DLFR-VAE: Dynamic Latent Frame Rate VAE for Video Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhihang Yuan, Siyuan Wang, Rui Xie, Hanling Zhang, Tongcheng Fang, Yuzhang Shang, Shengen Yan, Guohao Dai and 1 more
DLFR-VAE introduces a training-free framework that dynamically adjusts the latent frame rate within Variational Autoencoders based on video content complexity. This approach reduces the average number of latent space elements by 50% while maintaining reconstruction quality and achieving a 2x to 6x speedup in video generation without retraining existing generative models.
Key insights
- Real-world video content exhibits significant temporal non-uniformity, with high-motion segments containing more information than static scenes, and this variability is preserved in the latent space of pretrained VAEs.
- By applying principles similar to the Nyquist-Shannon sampling theorem, the latent-space frame rate can be adaptively lowered in segments with lower 'effective maximum frequency' without losing critical temporal details.
- A simple, computationally efficient proxy metric like mean inter-frame SSIM in the pixel domain can reliably estimate temporal content complexity and guide dynamic frame rate allocation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.