Reproduce “IFSeg: Image-free Semantic Segmentation via Vision-Language Model” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Sukmin Yun, Seong Hyeon Park, Paul Hongsuck Seo, Jinwoo Shin
Researchers introduce IFSeg, an approach for image-free semantic segmentation that enables models to segment images using only a list of target category names, without any training images or annotations. The method leverages pre-trained Vision-Language encoder-decoder models by generating artificial image-segmentation pairs from text, achieving 55.6 mIoU on unseen COCO Stuff categories and competitive performance on cross-dataset benchmarks.
Key insights
- Pre-trained Vision-Language (VL) models, especially encoder-decoder architectures, implicitly align visual and text tokens in a shared embedding space, containing sufficient knowledge to perform pixel-level tasks from text inputs.
- Semantic segmentation can be effectively reframed as a text generation problem where a VL decoder predicts a semantic category word for each image token.
- Artificial 'image' tokens, constructed by randomly sampling and arranging semantic category words, can serve as effective training data for adapting VL models to dense prediction tasks without real images or annotations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.