Reproduce “PANORAMA: Panoptic Grounded Captioning via Mask Proposal Selection” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Sara Pieri, Evangelos Kazakos, Shizhe Chen, Josef Sivic, Cordelia Schmid
PANORAMA combines dense free-form image captioning with pixel-level grounding by pairing a human-annotated benchmark, PanoCaps, with a mask-proposal selection architecture built on Qwen3-VL and SAM 3. On PanoCaps, its 4B model reaches 68.0 CAPTURE, 66.5 mIoU, and 45.6 generalized Panoptic Quality, while transferring competitively to referring segmentation and multi-region grounding tasks.
Key insights
- Separating referent identification from mask delineation is more effective than asking the VLM to generate masks directly: replacing proposal selection reduced validation gPQ from 42.1 to 34.6 and AP50 from 60.8 to 40.8.
- Human-curated dense annotations were more useful than larger, noisier automatically generated mixtures; PanoCaps-only training achieved gPQ 44.0 versus 37.9 for cleaned COCONut-PanCap and 18.8 for the GCG mixture.
- Phrase-conditioned proposals and score-aware matching are especially valuable when expressions refer to multiple similar entities or several regions, while auxiliary semantic-mask supervision mainly benefits single-region referring segmentation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.