Reproduce “SSP-SAM: SAM with Semantic-Spatial Prompt for Referring Expression Segmentation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Wei Tang, Xuejing Liu, Yanpeng Sun, Zechao Li
Researchers from Nanjing University of Science and Technology and the National University of Singapore developed SSP-SAM, a one-stage framework that enhances SAM's segmentation by incorporating CLIP's multimodal understanding via a Semantic-Spatial Prompt (SSP) encoder. The model achieves leading performance on classic Referring Expression Segmentation, Generalized RES, and open-vocabulary benchmarks, demonstrating improved precision at strict thresholds and efficient inference speeds.
Key insights
- An efficient, one-stage fusion of CLIP's multimodal understanding with SAM's segmentation is achievable via a specialized prompt encoder, avoiding extensive architectural modifications or heavy training.
- Deep semantic and spatial cue extraction from natural language, facilitated by combined visual and linguistic attention adapters (including Gaussian-like mapping), is crucial for precise language-guided segmentation.
- Incorporating an auxiliary bounding box regression task during pre-training and fine-tuning significantly leverages bounding box annotations to enhance prompt quality and overall segmentation performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.