Reproduce “RePre: Improving Self-Supervised Vision Transformer with Reconstructive Pre-training” on OpenResearch

About this paper

Luya Wang, Feng Liang, Yangguang Li, Honggang Zhang, Wanli Ouyang, Jing Shao

RePre integrates pixel-level reconstructive pre-training with existing contrastive learning frameworks for Vision Transformers, enabling them to learn both global and fine-grained local features. This hybrid approach consistently improves performance on ImageNet linear evaluation by up to 1.1% and significantly boosts downstream dense prediction tasks, surpassing supervised pre-training on COCO object detection by 1.2% mAP_bbox.

Key insights

• Reconstructing pixels effectively requires multi-hierarchy features, as single last-layer outputs are insufficient for generating fine-grained local information.
• A lightweight, convolutional decoder is optimal for the reconstruction task, enhancing local features without over-focusing on low-level semantics or adding excessive computational burden.
• An uncertainty weighting mechanism can dynamically balance the contributions of contrastive and reconstructive losses, optimizing joint learning without manual hyperparameter tuning.

Summary by alphaXiv.