Reproduce “Cut and Learn for Unsupervised Object Detection and Instance Segmentation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Xudong Wang, Rohit Girdhar, Stella X. Yu, Ishan Misra

CutLER provides an unsupervised framework for object detection and instance segmentation, using self-supervised Vision Transformer features, iterative mask generation, and a self-training loop with a DropLoss mechanism. It achieves state-of-the-art zero-shot performance on 11 diverse benchmarks, often doubling the AP of prior methods, and enhances label-efficient supervised learning.

Key insights

  • Self-supervised Vision Transformers, such as DINO, inherently capture object-centric information that can be effectively leveraged for initial object discovery.
  • Iteratively masking out discovered regions when applying Normalized Cuts allows for finding multiple distinct object instances within a complex scene, moving beyond single-object saliency.
  • A self-training loop with a specialized loss (DropLoss) enables the detector to refine coarse pseudo-masks and autonomously discover novel objects, leading to improved segmentation quality and quantity.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.