Reproduce “Multi-class Token Transformer for Weakly Supervised Semantic Segmentation” on OpenResearch

About this paper

Lian Xu, Wanli Ouyang, Mohammed Bennamoun, Farid Boussaid, Dan Xu

The Multi-class Token Transformer (MCTformer) introduces a framework for weakly supervised semantic segmentation that uses multiple class-specific tokens to generate accurate class-discriminative attention maps. By combining this with an attention refinement strategy and integrating a PatchCAM module, the method achieves 69.9% mIoU on the PASCAL VOC 2012 validation set and 42.0% mIoU on MS COCO 2014, producing more complete object localization and precise boundaries.

Key insights

• Introducing multiple class tokens into a Vision Transformer enables the model to generate accurate class-specific attention maps, overcoming the limitations of single-class token designs in weakly supervised semantic segmentation.
• Patch-to-patch attention maps inherently contain pixel affinity information that can be leveraged, without additional computation, to refine class-specific attention maps, improving object completeness and boundary precision.
• Fusing the multi-class token transformer attention with traditional PatchCAM maps provides complementary benefits, leading to superior object localization performance.

Summary by alphaXiv.