Reproduce “Siamese Masked Autoencoders” on OpenResearch

About this paper

Agrim Gupta, Jiajun Wu, Jia Deng, Li Fei-Fei

Researchers from Stanford and Princeton Universities developed Siamese Masked Autoencoders (SiamMAE), a self-supervised method for learning visual correspondence from videos by using an asymmetric masking strategy. SiamMAE achieves state-of-the-art performance among self-supervised methods on video object segmentation, video part segmentation, and pose tracking.

Key insights

• Asymmetric masking, where one frame is fully visible and the other is heavily masked (e.g., 95%), forces the model to learn explicit temporal correspondences between frames for successful reconstruction.
• A more challenging reconstruction task, specifically with high masking ratios and larger temporal gaps between sampled frames, leads to the learning of more robust and accurate visual correspondence representations.
• The model naturally develops an emergent ability to detect object boundaries, as evidenced by self-attention maps, despite not being explicitly trained for this task, indicating it learns object-centric representations from video motion.

Summary by alphaXiv.