Reproduce “Side Adapter Network for Open-Vocabulary Semantic Segmentation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Mengde Xu, Zheng Zhang, Fangyun Wei, Han Hu, Xiang Bai
Xu et al. from Huazhong University of Science and Technology and Microsoft Research Asia introduce the Side Adapter Network (SAN), a lightweight end-to-end framework for open-vocabulary semantic segmentation. The system adapts frozen, image-level pre-trained CLIP models to pixel-level tasks by generating CLIP-aware mask proposals and guiding CLIP's attention, achieving an average of +2.9 mIoU over prior methods with significantly fewer parameters and higher inference speed.
Key insights
- A lightweight, end-to-end trainable side network can effectively adapt a frozen image-level vision-language model for pixel-level tasks while preserving its broad open-vocabulary generalization capabilities.
- Guiding the attention mechanism of deeper CLIP layers with generated biases and introducing 'shadow [CLS] tokens' is crucial for bridging the granularity gap and enabling mask-specific recognition without fine-tuning CLIP's core parameters.
- Decoupling mask prediction and attention bias generation into separate heads improves performance, suggesting that the optimal regions for CLIP's recognition may differ from the mask region itself.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.