Reproduce “GroupViT: Semantic Segmentation Emerges from Text Supervision” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, Xiaolong Wang
GroupViT is a hierarchical grouping Vision Transformer that learns to perform semantic segmentation solely from image-level text supervision, eliminating the need for pixel-wise annotations. The model demonstrates zero-shot transfer capabilities, achieving 52.3% mIoU on PASCAL VOC 2012 and 22.4% mIoU on PASCAL Context, often outperforming or matching methods requiring pixel-level fine-tuning.
Key insights
- Semantic segmentation capabilities can emerge from purely image-level text supervision by integrating an explicit, hierarchical grouping mechanism within a Vision Transformer architecture.
- A "hard assignment" strategy in the grouping block, which exclusively assigns segment tokens to a single group, is crucial for learning distinct and well-differentiated semantic representations.
- Utilizing a multi-label contrastive loss, generated through prompt engineering, enhances the richness of textual supervision and significantly improves the model's ability to form semantically consistent groups.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.