Reproduce “VFM-VAE: Vision Foundation Models Can Be Good Tokenizers for Latent Diffusion Models” on OpenResearch

About this paper

Tianci Bi, Xiaoyi Zhang, Yan Lu, Nanning Zheng

VFM-VAE introduces a visual tokenizer for Latent Diffusion Models that directly integrates frozen Vision Foundation Model (VFM) encoders, overcoming representation degradation observed in distillation-based methods. This approach yields a robust and semantically rich latent space, enabling a 10x speedup in downstream diffusion model training convergence and achieving a gFID of 1.31 (w/ CFG) on ImageNet 256x256.

Key insights

• Directly using frozen VFM encoders preserves the robustness and semantic richness of VFM representations more effectively than distillation-based approaches, maintaining stable alignment under semantic-preserving perturbations.
• A carefully designed decoder is essential for reconstructing high-fidelity images from the semantically rich but spatially coarse representations provided by frozen VFM encoders.
• Combining VFM-VAE with explicit VFM alignment within the diffusion model itself (dual-side alignment) creates a synergistic effect, resulting in consistently strong and uniform VFM alignment across the entire generative pipeline.

Summary by alphaXiv.