Reproduce “A Simple Framework for Open-Vocabulary Segmentation and Detection” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hao Zhang, Feng Li, Xueyan Zou, Shilong Liu, Chunyuan Li, Jianfeng Gao, Jianwei Yang, Lei Zhang
OpenSeeD presents a unified encoder-decoder framework that learns open-vocabulary segmentation and detection by jointly training on separate bounding box and pixel-mask annotated datasets. The model achieves state-of-the-art performance in zero-shot transfer for both tasks and sets new benchmarks on panoptic and instance segmentation across various datasets, including COCO, ADE20K, and Cityscapes.
Key insights
- Jointly training on diverse, separate detection and segmentation datasets is an effective strategy for building a single, unified open-vocabulary model.
- Explicitly decoupling foreground (objects) and background (stuff) queries within a shared decoder mitigates task discrepancies, improving performance for both object detection and panoptic/semantic segmentation.
- Generating pseudo-masks for bounding box-only data via a conditioned mask decoding component effectively bridges the data granularity gap, allowing coarse box annotations to contribute to fine-grained segmentation learning.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.