Reproduce “SegCLIP: Patch Aggregation with Learnable Centers for Open-Vocabulary Semantic Segmentation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Huaishao Luo, Junwei Bao, Youzheng Wu, Xiaodong He, Tianrui Li
SegCLIP proposes a group-based method for open-vocabulary semantic segmentation that leverages pre-trained CLIP models, integrating a semantic group module, reconstruction loss, and superpixel-based KL loss for annotation-free training. The model achieves competitive performance, outperforming other group-based methods and approaching class-supervised models while segmenting arbitrary categories described by text.
Key insights
- Directly adapting pre-trained vision-language models with a specialized plugged module and self-supervised auxiliary losses enables efficient, annotation-free, open-vocabulary semantic segmentation.
- The reconstruction of masked patches based on dynamically formed irregular segments significantly refines the visual encoder's features for dense prediction tasks.
- Incorporating local visual coherence through superpixel pseudo-labels is effective in guiding the patch aggregation process to form consistent and accurate semantic regions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.