Reproduce “Repurposing Stable Diffusion Attention for Training-Free Unsupervised Interactive Segmentation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Markus Karmann, Onay Urfalioglu

Researchers at Vivo Tech Research GmbH developed M2N2, a training-free and unsupervised framework that repurposes Stable Diffusion's self-attention for interactive image segmentation. The method achieved state-of-the-art performance among unsupervised approaches, reducing NoC85 by 1.73 and NoC90 by 1.72 compared to previous methods like MIS on the DAVIS dataset, and proved competitive with some supervised models.

Key insights

  • Rich semantic information embedded within the self-attention mechanisms of large pre-trained generative models like Stable Diffusion can be directly leveraged for complex downstream tasks without further training.
  • Transforming raw attention maps into Markov-maps through doubly stochastic matrix conversion and iterative probability propagation significantly refines semantic coherence, reduces noise, and enhances instance-level distinction.
  • An adaptive, multi-component score function for dynamically truncating the Markov-maps is critical for achieving precise, instance-aware interactive segmentation with a nearest-neighbor classifier.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.