Reproduce “Context Autoencoder for Self-Supervised Representation Learning” on OpenResearch

About this paper

Xiaokang Chen, Mingyu Ding, Xiaodi Wang, Ying Xin, Shentong Mo, Yunhao Wang, Shumin Han, Ping Luo and 2 more

The Context Autoencoder (CAE), developed by a team from Peking University, University of Hong Kong, UC Berkeley, and Baidu, introduces a refined masked image modeling framework that explicitly predicts masked patch representations in the latent space. This design enables the learning of robust, general-purpose visual features, leading to improved performance on ImageNet classification, ADE20K semantic segmentation, and COCO object detection benchmarks.

Key insights

• Explicitly predicting masked patch representations in the encoded latent space, facilitated by a dedicated regressor and alignment loss, is critical for learning high-quality, semantically rich features.
• Masked Image Modeling (MIM) approaches, exemplified by CAE, encourage learning comprehensive features across the entire image, which benefits dense prediction tasks more than object-centric contrastive self-supervised learning methods.
• Decoupling the roles of the encoder (learning representations from visible patches), regressor (predicting masked latent representations), and decoder (reconstructing masked patches) leads to more robust and transferable visual representations.

Summary by alphaXiv.