Reproduce “BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizers” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhiliang Peng, Li Dong, Hangbo Bao, Qixiang Ye, Furu Wei
BEIT V2, developed by Microsoft Research and the University of Chinese Academy of Sciences, advances masked image modeling by integrating a vector-quantized knowledge distillation (VQ-KD) algorithm to generate semantic-aware visual tokens. This approach achieved 85.5% top-1 accuracy on ImageNet-1K for ViT-B/16 (1600 epochs) and 56.7% mIoU on ADE20K for ViT-L/16 (1600 epochs), outperforming prior self-supervised methods.
Key insights
- Aligning the MIM reconstruction target with high-level semantic concepts, analogous to successful Masked Language Modeling in NLP, yields superior and more transferable visual representations.
- A visual tokenizer trained with knowledge distillation from a semantic-rich teacher model (like CLIP) effectively discretizes semantic space, capturing concepts that are robust to low-level image details.
- Explicitly encouraging global information aggregation during patch-level pretraining significantly enhances the quality of the global `[CLS]` token representation, improving performance on image-level tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.