Reproduce “Large Motion Video Autoencoding with Cross-modal Video VAE” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yazhou Xing, Yang Fei, Yingqing He, Jingye Chen, Jiaxin Xie, Xiaowei Chi, Qifeng Chen
Researchers at The Hong Kong University of Science and Technology developed a Cross-modal Video VAE designed for high-fidelity video encoding and decoding, particularly for videos with large motions. It mitigates issues like motion ghosting and detail distortion through a two-stage spatiotemporal architecture, cross-modal text guidance, and joint image-video training, achieving state-of-the-art reconstruction quality.
Key insights
- Decoupling spatiotemporal compression into a temporal-aware spatial stage and a sequential temporal stage effectively preserves spatial details while recovering complex motions.
- Integrating cross-modal information from text descriptions via attention mechanisms can semantically guide the autoencoding process, improving detail recovery and reconstruction fidelity.
- Jointly training the VAE on both image and video data significantly boosts video reconstruction performance and versatility, allowing the model to leverage diverse datasets.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.