Reproduce “Diffuse, Attend, and Segment: Unsupervised Zero-Shot Segmentation using Stable Diffusion” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Junjiao Tian, Lavisha Aggarwal, Andrea Colaco, Zsolt Kira, Mar Gonzalez-Franco
DiffSeg leverages the self-attention mechanisms of a pre-trained Stable Diffusion model to perform unsupervised zero-shot image segmentation on real images. The method substantially improves pixel accuracy by 26% and mIoU by 17% over prior unsupervised zero-shot techniques on COCO-Stuff-27 and generalizes to diverse domains like sketches, satellite, and medical images.
Key insights
- Large-scale pre-trained generative models, specifically Stable Diffusion, implicitly learn rich object-level semantic information within their unconditional self-attention layers during their denoising process.
- This latent object understanding can be extracted directly from the model to perform high-quality image segmentation in an unsupervised and zero-shot manner, without requiring explicit annotations or text prompts.
- The aggregation of multi-resolution self-attention maps and an iterative merging process effectively cluster these inherent attention patterns into coherent object proposals.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.