Reproduce “Advancing Visual Large Language Model for Multi-granular Versatile Perception” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Wentao Xiang, Haoxian Tan, Cong Wei, Yujie Zhong, Dengjie Li, Yujiu Yang
MVP-LM is a Visual Large Language Model that unifies diverse visual perception tasks, including bounding box and pixel-level mask predictions from both word-based and sentence-based instructions, within a single framework. The model demonstrates robust generalization in open-set scenarios and achieves competitive performance, notably outperforming larger VLLMs in referring expression comprehension with a relatively compact 1.3 billion parameter language model.
Key insights
- Joint training across diverse perception datasets significantly boosts sentence-based task performance while maintaining or improving word-based tasks, empirically validating the unified multi-task approach.
- A "thinking-then-perceiving" paradigm, by integrating image captions into the LLM's response, improves performance across both sentence-level and word-level perception by encouraging deeper visual context understanding.
- Dynamic query generation, leveraging LLM's high-level understanding and language-guided visual features, is crucial for versatile and precise perception across multi-target tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.