Reproduce “Multi-modal Instruction Tuned LLMs with Fine-grained Visual Perception” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Junwen He, Yifan Wang, Lijun Wang, Huchuan Lu, Jun-Yan He, Jin-Peng Lan, Bin Luo, Xuansong Xie
A general multi-modal large language model, AnyRef, achieves fine-grained pixel-level object segmentation and referring expression generation from diverse inputs, including text, bounding boxes, visual examples, and audio. It demonstrates competitive performance across various multi-modal referring tasks, providing a unified framework for flexible human-AI interaction.
Key insights
- The refocusing mechanism effectively leverages internal LLM attention to implicitly supervise language generation with pixel-level data, improving regional understanding with minimal computational overhead.
- Converting diverse multi-modal references into a unified token-based representation allows a single LLM to process and reason about various input types without modality-specific architectural changes.
- Comprehensive training on diverse datasets (semantic, instance, referring expression, image-level, and audio-visual segmentation data) is crucial for building a generalist MLLM with broad capabilities.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.