Reproduce “Prompting Diffusion Models for Zero-Shot Instance Segmentation” on OpenResearch

About this paper

Irem Zeynep Alagöz, Nils Morbitzer, Andrea Ramazzina, Nassir Navab, Federico Tombari, Stefano Gasperini

Researchers developed Prompt2Seg, a framework that adapts a frozen diffusion segmentation model for interactive instance segmentation by integrating sparse spatial prompts via a ControlNet-style branch. This approach demonstrates robust zero-shot generalization, outperforming the Segment Anything Model in various challenging "long-tail" visual domains while being trained solely on synthetic data.

Key insights

• Integrating spatial prompts directly into the diffusion decoding process, rather than using them for post-hoc filtering, enables generative models to produce more semantically coherent and user-aligned instance segments.
• The rich visual priors acquired by large-scale diffusion models from self-supervised image reconstruction contribute to exceptional zero-shot generalization capabilities, even when fine-tuned on limited synthetic data.
• A ControlNet-style architecture effectively allows for explicit spatial control over a frozen diffusion backbone, transforming it into an interactive segmentation tool without retraining the entire large model.

Summary by alphaXiv.