Reproduce “AVTok: 1D Unified Tokenization for Holistic Audio-Video Generation” on OpenResearch

About this paper

Kien T. Pham, I Chieh Chen, Qifeng Chen, Long Chen

Researchers from The Hong Kong University of Science and Technology introduce AVTok, a novel dual-stream transformer-based architecture that achieves unified 1D tokenization for both audio and video. This approach enables joint encoding of audio and video into a single compact latent representation, leading to competitive performance in reconstruction and improving the quality and coherence of content generated by downstream audio-video models.

Key insights

• The dual-stream architecture, by using shared encoder-decoder parameters alongside modal-specific queries and normalization, effectively balances modality-specific processing with implicit cross-modal information fusion.
• The hierarchical VFAL training strategy is crucial for successfully training unified audio-video models, effectively managing information imbalance between video's high density and audio's temporal nature.
• Explicit representation alignment using a pretrained audio-visual model significantly enhances the semantic correspondence captured within the unified latent space, improving overall reconstruction and generation quality.
• Integrating a lightweight AR prior during training is essential for shaping the discrete latent space to be highly compatible with autoregressive generative models, optimizing their performance in downstream synthesis tasks.

Summary by alphaXiv.