Reproduce “Empowering Segmentation Ability to Multi-modal Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yuqi Yang, Peng-Tao Jiang, Jing Wang, Hao Zhang, Kai Zhao, Jinwei Chen, Bo Li
Researchers at vivo Mobile Communication Co., Ltd. developed LLaVASeg, a framework that integrates segmentation capabilities into multi-modal large language models (MLLMs) using a Chain-of-Thought prompting strategy and a multi-scale segmentation network. This approach preserves the MLLM's original conversational and reasoning abilities, outperforming prior methods like LISA on ReasonSeg gIoU by 1.8% and showing significantly higher ROUGE-L and CIDEr scores for dialogue quality.
Key insights
- Decoupling MLLM reasoning from segmentation execution, achieved by freezing the MLLM's parameters, is crucial for preventing the degradation of its original conversational capabilities.
- Explicitly guiding the MLLM through a Chain-of-Thought process to generate detailed visual attributes (e.g., color, shape, relative position) significantly enhances the precision of reasoning segmentation.
- Integrating language-aware adapters at multiple scales within the segmentation network allows for effective fusion of textual attributes with visual features across different semantic levels, improving segmentation accuracy.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.