Reproduce “NExT-Chat: An LMM for Chat, Detection and Segmentation” on OpenResearch

About this paper

Ao Zhang, Yuan Yao, Wei Ji, Zhiyuan Liu, Tat-Seng Chua

NExT-Chat introduces "pix2emb," a novel location modeling paradigm for Large Multimodal Models (LMMs) that represents object locations as continuous embeddings, enabling flexible output of both bounding boxes and segmentation masks within a unified conversational framework. The model demonstrates strong performance in referring expression segmentation, region captioning, and hallucination reduction, often surpassing prior LMMs, particularly with efficient segmentation training.

Key insights

• Representing object location as continuous embeddings (pix2emb) significantly enhances flexibility, allowing LMMs to generate diverse output formats like bounding boxes and precise segmentation masks within a unified architecture.
• Leveraging pre-trained models like SAM for segmentation, combined with a targeted, efficient training strategy, enables LMMs to acquire strong segmentation capabilities with substantially less segmentation training data compared to prior methods.
• Integrating robust input and output location modeling within an LMM framework fosters unified multimodal conversations, enables more accurate grounded reasoning, and effectively reduces object hallucination in generated text.

Summary by alphaXiv.