Reproduce “Rethinking Semantic Segmentation: A Prototype View” on OpenResearch

About this paper

Tianfei Zhou, Wenguan Wang, Ender Konukoglu, Luc Van Gool

This work re-conceptualizes semantic segmentation by adopting a nonparametric, prototype-based approach that represents classes with multiple non-learnable prototypes derived from pixel sub-clusters. It achieves consistent mIoU improvements on datasets like ADE20K (e.g., 0.8% gain with SegFormer MiT-B4) and demonstrates enhanced scalability for large-vocabulary scenarios by eliminating learnable prototype parameters.

Key insights

• Existing state-of-the-art segmentation methods, regardless of architecture, can be viewed as parametric models using learnable class prototypes.
• Representing classes with multiple, non-learnable prototypes derived from data effectively captures rich intra-class variations, which single parametric prototypes cannot achieve.
• Directly optimizing the pixel embedding space through metric learning with non-learnable prototypes leads to more organized, compact, and discriminative feature representations than relying solely on classification loss.

Summary by alphaXiv.