Reproduce “MCTformer+: Multi-Class Token Transformer for Weakly Supervised Semantic Segmentation” on OpenResearch

About this paper

Lian Xu, Mohammed Bennamoun, Farid Boussaid, Hamid Laga, Wanli Ouyang, Dan Xu

MCTformer+ presents a Vision Transformer-based framework for weakly supervised semantic segmentation that generates high-quality, class-specific pseudo-labels. The model achieves state-of-the-art performance, with a 74.0% mIoU on the PASCAL VOC 2012 validation set and 73.6% on the test set, by utilizing multiple class tokens, a contrastive class token module, and global weighted ranking pooling.

Key insights

• A multi-class token architecture in transformers can inherently generate class-specific localization maps, directly addressing the limitations of a single, class-agnostic class token.
• Applying a contrastive loss to class tokens throughout all transformer layers significantly improves their distinctiveness and ability to produce non-overlapping class attention maps.
• The transformer's intrinsic patch-to-patch attention can be effectively re-purposed as a 'free' affinity matrix to refine localization maps, improving completeness and boundary precision without extra supervision.

Summary by alphaXiv.