Reproduce “DeCo-VAE: Learning Compact Latents for Video Reconstruction via Decoupled Representation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xiangchen Yin, Jiahui Yuan, Zhangchi Hu, Wenzhang Sun, Jie Chen, Xiaozhen Qiao, Hao Li, Xiaoyan Sun
Researchers from the University of Science and Technology of China (USTC) and Li Auto Inc. developed DeCo-VAE, a video Variational Autoencoder that explicitly decouples video content into keyframe, motion, and residual components to learn a more compact and interpretable latent representation. This approach achieves state-of-the-art video reconstruction on WebVid and Kinetics-400 datasets with a lightweight 62M parameter model, also improving downstream video generation performance in diffusion models.
Key insights
- Decomposing video into keyframe, motion, and residual components, akin to traditional codecs, enables more compact and interpretable latent representations for deep learning models.
- Employing dedicated encoders for each decoupled component is crucial for preventing feature entanglement and precisely capturing distinct static and dynamic video information.
- A two-phase decoupled adaptation strategy during training ensures stable learning and optimized modeling of both static appearance and dynamic temporal features.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.