Reproduce “Emerging Properties in Self-Supervised Vision Transformers” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, Armand Joulin

DINO trains Vision Transformers without labels by matching a student’s outputs to those of a slowly updated teacher across different views of each image. The teacher uses centered, sharpened probability distributions, and the student learns through cross-entropy. DINO features support object-layout information in attention maps and strong nearest-neighbor classification, but do not establish general-purpose segmentation or uniquely DINO properties.

Key insights

  • Self-supervised learning is critical for unlocking unique capabilities in Vision Transformers (ViTs) not observed with supervised training, particularly regarding spatial understanding.
  • DINO-trained ViTs intrinsically develop emergent semantic segmentation capabilities, visible in their self-attention maps, without explicit pixel-level supervision.
  • The momentum teacher within DINO consistently outperforms the student during training, acting as a 'better role model' and guiding the learning process.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.