Reproduce “Attention Guided Alignment in Efficient Vision-Language Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Shweta Mahajan, Hoang Le, Hyojin Park, Farzad Farhadzadeh, Munawar Hayat, Fatih Porikli

Qualcomm AI Research developed Attention-Guided Efficient Vision-Language Models (AGE-VLM), which enhance visual grounding and reduce object hallucination in efficient VLMs by integrating SAM-distilled spatial knowledge into interleaved cross-attention layers of a 1B LLaMA model. The approach achieved 87.34% accuracy on the POPE benchmark and notable improvements on CV-Bench for 2D/3D understanding compared to prior efficient methods.

Key insights

  • Analysis of existing efficient VLMs showed inadequate visual-textual alignment, with models either failing to distinguish relevant pairs or exhibiting indiscriminately high similarity, indicating a root cause of hallucination.
  • Knowledge distillation from SAM, explicitly aligning cross-attention with segmentation masks, effectively imparts fine-grained spatial grounding to efficient VLMs.
  • The spatial grounding acquired during pre-training is sufficiently robust, allowing for competitive performance even without continued grounding loss during instruction fine-tuning.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.