Reproduce “Q-GroundCAM: Quantifying Grounding in Vision Language Models via GradCAM” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Navid Rajabi, Jana Kosecka
George Mason University researchers introduced Q-GroundCAM, a comprehensive framework that quantitatively evaluates how well vision-language models (VLMs) ground language to visual content by extending GradCAM. The framework, applied to models like BLIP, CLIP, and ALBEF, demonstrated that ALBEF with Attention Mask Consistency, fine-tuned with bounding box-level supervision, achieves superior grounding performance compared to other VLMs, even larger ones.
Key insights
- Q-GroundCAM metrics provide more nuanced differentiation between Vision-Language Models (VLMs) than the traditional Pointing Game alone.
- The Inside/Outside (IO) ratio is a strong indicator of grounding performance, correlating with Pointing Game accuracy but offering a more discriminative measure.
- Model size does not guarantee superior grounding, as BLIP Large did not consistently outperform BLIP Base, while fine-tuning with bounding box-level supervision (e.g., ALBEF AMC) generally improves grounding capability.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.