Reproduce “Turbo-VAED: Fast and Stable Transfer of Video-VAEs to Mobile Devices” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ya Zou, Jingfeng Yao, Siyuan Yu, Shuai Zhang, Wenyu Liu, Xinggang Wang
Turbo-VAED develops a low-cost, efficient solution for transferring large video Variational AutoEncoders (VAEs) to mobile devices, achieving real-time 720p video decoding on an iPhone 16 Pro at up to 38.1 FPS while maintaining high reconstruction and generation quality. The framework significantly reduces VAE parameters by up to 82.5% and offers substantial inference speedups on both GPUs and mobile platforms.
Key insights
- Low-resolution layers of VAE decoders exhibit higher parameter redundancy, making them ideal targets for parameter-efficient convolutions to achieve substantial model compression with minimal quality loss.
- The 3D pixel shuffle operation, while effective on GPUs, is a primary latency bottleneck on mobile devices due to poor kernel compatibility, necessitating a custom decoupled approach for efficient mobile upsampling.
- Intermediate feature alignment during decoder-only distillation significantly accelerates training convergence (2.2x speedup) and improves reconstruction quality, especially when aligning features from multiple low-to-mid resolution layers.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.