Reproduce “DPSeg: Dual-Prompt Cost Volume Learning for Open-Vocabulary Semantic Segmentation” on OpenResearch

About this paper

Ziyu Zhao, Xiaoguang Li, Linjia Shi, Nasrin Imanpour, Song Wang

DPSeg introduces a dual-prompt framework for open-vocabulary semantic segmentation, combining visual and text prompts to mitigate the image-text domain gap. The model employs a cost volume-guided decoder with multi-level feature guidance and a semantic-guided prompt refinement strategy, achieving new state-of-the-art performance across various benchmarks by improving accuracy and fine-grained detail segmentation.

Key insights

• Visual prompts generated from text exhibit stronger feature alignment with target images and produce sharper semantic contours compared to text prompts alone, effectively bridging the modality gap.
• The synergistic combination of both visual and text prompts yields a more robust and semantically rich representation for pixel-level classification.
• Directly computing multi-scale visual cost volumes at intermediate decoder stages is crucial for preserving fine-grained details and improving segmentation of small objects, outperforming simple upsampling of a single cost volume.
• Refining generic prompts with scene-specific visual information derived from initial segmentation predictions significantly enhances semantic alignment and segmentation precision during inference.

Summary by alphaXiv.