Reproduce “Improving Masked Autoencoders by Learning Where to Mask” on OpenResearch

About this paper

Haijian Chen, Wendong Zhang, Yunbo Wang, Xiaokang Yang

Researchers at Shanghai Jiao Tong University developed AutoMAE, an end-to-end differentiable framework for Masked Autoencoders that learns an adaptive masking strategy instead of relying on random masking. AutoMAE improves learned visual representations, outperforming standard MAE on various downstream tasks and demonstrating particular strength in low-data finetuning scenarios.

Key insights

• Empirical evidence confirms that an adaptive masking strategy, focusing on object-centric regions, improves representation learning but must balance information gain with reconstruction task difficulty.
• Adversarial training, using 'pseudo-real' object-centric masks, can effectively guide a mask generator to identify semantically meaningful regions without explicit supervision.
• Joint, end-to-end optimization of the masking strategy with the MAE's reconstruction task is crucial for the masking process to adapt optimally to the specific learning dynamics.

Summary by alphaXiv.