Reproduce “CLIPer: Hierarchically Improving Spatial Representation of CLIP for Open-Vocabulary Semantic Segmentation” on OpenResearch

About this paper

Lin Sun, Jiale Cao, Jin Xie, Xiaoheng Jiang, Yanwei Pang

CLIPer introduces a training-free framework for open-vocabulary semantic segmentation that hierarchically enhances CLIP's spatial representation by fusing early-layer attention and refining details with Stable Diffusion. The method achieves state-of-the-art mIoU across seven datasets, for instance, improving VOC mIoU to 69.8% with a ViT-L/14 backbone, a 9.2% increase over previous leading methods.

Key insights

• Early layers of CLIP's Vision Transformer encoder contain rich and coherent spatial structural information that can be effectively leveraged for better spatial representation.
• Integrating hierarchical attention from CLIP's early layers significantly improves global spatial coherence in segmentation without additional training.
• Stable Diffusion's self-attention mechanisms, even with a null prompt, effectively capture fine-grained local details, which can be synergistically used to compensate for the inherent coarseness of CLIP's patch-level outputs.

Summary by alphaXiv.