Reproduce “When Language Overwrites Vision: Over-Alignment and Geometric Debiasing in Vision-Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Harshvardhan Saini, Samyak Jha, Yiming Tang, Dianbo Liu
The authors reduce object hallucinations in decoder-based vision-language models by removing visual-feature projections onto top principal components of caption embeddings, either at inference or during fine-tuning. On LLaVA-1.5-7B, training-free projection lowers CHAIR_s from 52.80 to 45.60; POPE improves from 82.8 to 83.8, while Recall falls from 77.90 to 75.11, so gains are not uniform.
Key insights
- Decoder-based VLMs undergo "geometric over-alignment," where visual embeddings are over-compressed into a linguistically biased text manifold, injecting a universal statistical linguistic bias.
- This linguistic bias is concentrated in the top principal components of the latent space, systematically overshadowing and degrading the decodability of fine-grained visual information.
- Explicitly projecting out this identified textual bias from visual representations can unmask the underlying orthogonal visual signal, leading to more visually grounded outputs.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.