Reproduce “GSVA: Generalized Segmentation via Multimodal Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhuofan Xia, Dongchen Han, Yizeng Han, Xuran Pan, Shiji Song, Gao Huang
GSVA, developed by Tsinghua University, introduces a multimodal large language model that extends referring expression segmentation to handle instructions involving multiple distinct objects and explicitly recognize non-existent objects. The model sets a new state-of-the-art on the gRefCOCO dataset, achieving 70.04% gIoU on the validation set and significantly improving accuracy for rejecting empty targets.
Key insights
- Enabling MLLMs to predict multiple [SEG] tokens, each guided by its referring expression, is crucial for accurate multi-target segmentation.
- Introducing a dedicated [REJ] token allows MLLMs to explicitly and effectively handle references to non-existent objects, preventing erroneous mask generation.
- Weight sharing among multiple [SEG] tokens is important for stable training and avoiding performance degradation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.