Reproduce “DiCLIP: Diffusion Model Enhances CLIP's Dense Knowledge for Weakly Supervised Semantic Segmentation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Zhiwei Yang, Pengfei Song, Yucong Meng, Kexue Fu, Shuo Wang, Zhijian Song

DiCLIP, developed by researchers from Fudan University and affiliated institutions, improves weakly supervised semantic segmentation by integrating Stable Diffusion's spatial consistency and semantic richness to enhance CLIP's dense knowledge. This framework achieved 78.8% mIoU on PASCAL VOC 2012 and 48.7% mIoU on MS COCO 2014, while significantly reducing training time.

Key insights

  • Generative diffusion models can serve as external knowledge sources to inject reliable spatial consistency and semantic purity into CLIP's representations, compensating for its inherent limitations in dense prediction.
  • Refining CLIP's attention maps with information from SD's attention mechanisms leads to more discriminative visual features and improved spatial awareness for pixel-level tasks.
  • A dynamically updated cache of visual prototypes, initialized from SD-generated data, effectively augments text embeddings and allows for a learnable, robust patch-wise knowledge retrieval for superior CAM generation.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.