Reproduce “GiVE: Guiding Visual Encoder to Perceive Overlooked Information” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Junjie Li, Jianghong Ma, Xiaofeng Zhang, Yuhang Li, Jianyang Shi
Researchers from Harbin Institute of Technology and City University of Hong Kong introduce GiVE, a method to guide visual encoders in perceiving overlooked image information for Multimodal Large Language Models (MLLMs). The approach significantly improves image classification F1 scores on LVIS datasets, achieving gains of 348.7% on LVIS_f and 553.5% on LVIS_a when integrated with CLIP-ViT, and enhances image-text retrieval R@1 on MOInst by 282.9% for Image Text and 451.9% for Text Image.
Key insights
- Injecting semantic instructions directly into the visual encoder via a lightweight AG-Adapter is more effective for MLLM performance than applying instructions in a post-processing stage, as seen in models like InstructBLIP.
- The combination of object-focused contrastive (OITC, OIIC) and discriminative (OID) loss functions is crucial for guiding the visual encoder to comprehensively perceive both salient and non-salient visual information.
- Utilizing abstract, short object prompts for instruction is more effective for semantic guidance than detailed captions, preventing the visual encoder from over-relying on textual input and overlooking visual details.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.