Reproduce “Contrastive Learning Rivals Masked Image Modeling in Fine-tuning via Feature Distillation” on OpenResearch

About this paper

Yixuan Wei, Han Hu, Zhenda Xie, Zheng Zhang, Yue Cao, Jianmin Bao, Dong Chen, Baining Guo

This work from Tsinghua University and Microsoft Research Asia introduces Feature Distillation (FD), a post-processing method that significantly enhances the fine-tuning performance of various pre-trained vision models, including contrastive learning and image-text models. FD enables these models to rival or surpass Masked Image Modeling (MIM) approaches by making their representations more "optimization friendly," leading to +1.0% to +2.0% accuracy gains on ImageNet-1K and establishing new state-of-the-art results on several benchmarks.

Key insights

• MIM's fine-tuning advantage stems from learning "optimization friendly" representations, a property that can be induced in other pre-trained models through Feature Distillation.
• FD promotes more diverse attention heads and enhances translational invariance (reliance on relative spatial cues) in the student model's representations, which contributes to better fine-tuning.
• Models processed with FD exhibit flatter loss landscapes during fine-tuning, indicating easier optimization and improved generalization performance.

Summary by alphaXiv.