Reproduce “Dataset Diffusion: Diffusion-based Synthetic Dataset Generation for Pixel-Level Semantic Segmentation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Quang Nguyen, Truong Vu, Anh Tran, Khoi Nguyen
VinAI Research developed Dataset Diffusion, a framework that automatically generates photo-realistic images with pixel-level semantic segmentation masks from text prompts, eliminating the need for manual annotation. The method leverages a pre-trained Stable Diffusion model's internal attention mechanisms and achieves 64.8 mIoU on the synth-VOC benchmark, outperforming concurrent methods and providing a strong baseline for models trained on real data.
Key insights
- Pre-trained text-to-image diffusion models possess rich internal spatial and semantic information within their attention maps that can be effectively leveraged for pixel-level segmentation.
- Explicitly guiding the diffusion model with combined descriptive captions and precise class labels (class-prompt appending) is crucial for generating diverse, multi-object scenes with accurate masks.
- Refining coarse cross-attention maps with self-attention (self-attention exponentiation) is essential for achieving precise object boundaries and high-quality pixel-level segmentation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.