Reproduce “Weakly-Supervised Semantic Segmentation with Image-Level Labels: from Traditional Models to Foundation Models” on OpenResearch

About this paper

Zhaozheng Chen, Qianru Sun

The paper comprehensively evaluates the integration of visual foundation models, specifically the Segment Anything Model (SAM), into weakly-supervised semantic segmentation (WSSS) using image-level labels. It demonstrates that SAM-generated pseudo masks achieve significantly higher quality, with SAM (text input) outperforming traditional state-of-the-art methods by 12.7% mIoU on VOC, and SAM (zero-shot) surprisingly surpassing a fully-supervised DeepLabV2 baseline on MS COCO's validation set (54.6% vs. 53.5% mIoU).

Key insights

• Foundation models like SAM generate significantly higher quality pseudo-masks compared to traditional CAM-based methods, indicating a substantial leap in object localization capabilities.
• SAM, particularly in a zero-shot inference pipeline, can achieve segmentation performance that not only rivals but, in some cases, surpasses fully-supervised models on complex datasets like MS COCO by directly leveraging its pre-trained generalization.
• Failure cases primarily stem from limitations in upstream models (tagging/grounding) and reveal interesting discrepancies between SAM's object understanding and specific dataset annotation conventions, suggesting SAM can sometimes provide a more semantically complete segmentation.

Summary by alphaXiv.