Reproduce “Suppressing Non-Semantic Noise in Masked Image Modeling Representations” on OpenResearch

About this paper

Martine Hjelkrem-Tan, Marius Aasan, Rwiddhi Chakraborty, Gabriel Y. Arteaga, Changkyu Choi, Adín Ramírez Rivera

Researchers at the University of Oslo analyzed how Masked Image Modeling (MIM) objectives bias Vision Transformer representations toward non-semantic positional noise, proposing a Semantic Invariance (SI) Score to quantify this effect. They developed Semantically Orthogonal Artifact Projection (SOAP), a training-free post-hoc method that consistently improved zero-shot performance across various MIM models on salient segmentation and kNN semantic segmentation tasks.

Key insights

• Masked Image Modeling (MIM) objectives inherently bias Vision Transformers towards encoding non-semantic positional noise, which persists into deeper layers and is quantifiable by the Semantic Invariance Score.
• Leading principal components in MIM models exhibit strong positional biases and high semantic invariance, a characteristic consistently absent in non-MIM self-supervised approaches.
• A post-hoc suppression of these semantically invariant components significantly enhances the semantic fidelity and generalization capability of MIM representations for downstream tasks.

Summary by alphaXiv.