Reproduce “Improved Video VAE for Latent Video Diffusion Model” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Pingyu Wu, Kai Zhu, Yu Liu, Liming Zhao, Wei Zhai, Yang Cao, Zheng-Jun Zha
Researchers from USTC and Tongyi Lab developed an Improved Video VAE (IV-VAE) that optimizes latent video diffusion models by refining image VAE initialization and addressing inter-frame inconsistencies in causal convolutions. This approach achieves state-of-the-art video reconstruction and generation, demonstrating superior temporal consistency and performance across varied resolutions, while also introducing the MotionHD dataset for robust evaluation.
Key insights
- Initializing a video VAE with the same latent channel dimensions from a pre-trained image VAE actually hinders temporal compression learning, suggesting a need for a more balanced spatial-temporal prior integration.
- The unidirectional information flow of standard 3D causal convolution within frame groups causes inter-frame performance imbalance and flickering, which can be mitigated by allowing bidirectional interaction within groups while preserving causality across groups.
- Expanding the model's receptive field through methods like atrous convolutions and strategically placed attention modules is crucial for effectively perceiving and encoding complex motion patterns, especially in high-resolution video.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.