Reproduce “VFRTok: Variable Frame Rates Video Tokenizer with Duration-Proportional Information Assumption” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tianxiong Zhong, Xingye Tian, Boyuan Jiang, Xuebo Wang, Xin Tao, Pengfei Wan, Zhiwei Zhang
VFRTok is introduced as a variable frame rate video tokenizer for Latent Diffusion Models, operating on a duration-proportional information assumption to reduce computational costs. This tokenizer generates high-fidelity video efficiently across diverse frame rates while using substantially fewer latent tokens and significantly lowering computational overhead.
Key insights
- Video information capacity is better represented by its duration rather than its frame count, particularly at higher frame rates, enabling more efficient compression.
- Decoupling positional encoding from content modeling in Transformer attention mechanisms improves content-awareness and the compactness of latent representations.
- Asymmetric frame rate training is essential for a video tokenizer to learn continuous spatio-temporal representations and effectively generalize across diverse and arbitrary frame rates.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.