Reproduce “Mechanisms of Object Localization in Vision-Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Timothy Schaumlöffel, Martina G. Vilas, Gemma Roig
The authors investigate how vision-language models turn object information into bounding boxes. They enlarge object-aligned visual token regions with copied features and compare predictions against boxes of different extents. In LLaVA-7B, predictions match the once-expanded box at 44.0%, versus 24.6% and 24.8% for other boxes. Shuffling barely changes LLaVA localization but reduces InternVL3.5-8B localization, limiting claims across architectures.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.