Reproduce “MacTok: Robust Continuous Tokenization for Image Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hengyu Zeng, Xin Gao, Guanghao Li, Yuxiang Yan, Jiaoyang Ruan, Junpeng Ma, Haoyu Albert Wang, Jian Pu
MacTok, a continuous tokenizer from Fudan University, robustly learns compact and semantically rich 1D latent representations by addressing posterior collapse in KL-based VAEs. It achieves a new state-of-the-art gFID of 1.52 on ImageNet 512x512 with 128 tokens and competitive reconstruction fidelity while using significantly fewer tokens.
Key insights
- Image-level masking, particularly with semantic guidance, fundamentally mitigates posterior collapse by forcing the latent space to retain essential and semantically discriminative information.
- Multi-level representation alignment with pre-trained vision models (DINOv2) is crucial for organizing the latent space, enabling semantically similar concepts to cluster even with high compression.
- It is possible to achieve state-of-the-art generative performance with significantly fewer tokens (e.g., 64 or 128) by learning robust, semantically rich continuous latent representations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.