Reproduce “An Experimental Study on Exploring Strong Lightweight Vision Transformers via Masked Image Modeling Pre-Training” on OpenResearch

About this paper

Jin Gao, Shubo Lin, Shaoru Wang, Yutong Kou, Zeming Li, Liang Li, Congxuan Zhang, Xiaoqin Zhang and 2 more

Researchers at the Chinese Academy of Sciences and Megvii Research developed a decoupled distillation strategy (D2-MAE) that enhances the pre-training of simple lightweight Vision Transformers (ViTs) using Masked Image Modeling (MIM). This method transfers high-level semantic knowledge from a large teacher model, enabling lightweight ViTs to achieve state-of-the-art performance (e.g., 79.4% Top-1 accuracy on ImageNet-1K for ViT-Tiny) and improved transferability across diverse downstream tasks without requiring complex architectural modifications.

Key insights

• Standard MIM pre-training introduces a locality inductive bias in lightweight ViTs, leading to more local and concentrated attention patterns, which benefits fine-grained ImageNet-1K classification but hinders transferability to diverse, data-insufficient downstream tasks.
• Lightweight ViTs pre-trained with base MIM effectively learn low-level features in lower layers but struggle to acquire abstract, high-level semantics in deeper layers, which contrasts with the semantic learning observed in larger MAE-trained models.
• Decoupling the pixel reconstruction objective from high-level semantic distillation in the top layers allows lightweight ViTs to learn more global, recognition-aligned attention patterns in their higher layers, resolving conflicts between pixel-level reconstruction and abstract feature learning.

Summary by alphaXiv.