Reproduce “VidTok: A Versatile and Open-Source Video Tokenizer” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Anni Tang, Tianyu He, Junliang Guo, Xinle Cheng, Li Song, Jiang Bian
VidTok is an open-source, versatile video tokenizer developed by Microsoft Research, Shanghai Jiao Tong University, and Peking University, achieving state-of-the-art performance in video reconstruction for both continuous and discrete tokenization. It features a hybrid spatial-temporal architecture, integrates Finite Scalar Quantization (FSQ) for stable discrete representations, and employs an efficient two-stage training strategy.
Key insights
- A hybrid 2D/3D architectural approach effectively balances reconstruction quality with computational efficiency in video tokenization, outperforming both fully 3D and fully 2D convolutional designs.
- Finite Scalar Quantization (FSQ) provides superior training stability, nearly 100% codebook utilization, and better reconstruction fidelity for discrete video tokenization compared to traditional VQ and LFQ methods.
- Implementing a two-stage training strategy can halve GPU hours while maintaining performance, and training with reduced frame rates (e.g., 3 FPS) improves the model's ability to represent motion dynamics.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.