Reproduce “MDSAM:Memory-Driven Sparse Attention Matrix for LVLMs Hallucination Mitigation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shuaiye Lu, Linjiang Zhou, Xiaochuan Shi
Researchers at Wuhan University developed Memory-Driven Sparse Attention Matrix (MDSAM), a training-free method to mitigate hallucinations in Large Vision-Language Models (LVLMs) by dynamically refining attention to image tokens during the decoding process. This approach consistently reduced hallucination rates on CHAIR and MMHal-Bench, improved F1 scores on POPE, and increased overall performance on MME across various LVLMs.
Key insights
- Hallucinations in LVLMs are often caused by 'noise' within each layer's attention mechanism, leading to both under- and over-attention to visual features.
- Dynamically capturing and refining attention to image tokens at each decoder layer during inference, without retraining, can effectively guide the model towards more factually grounded outputs.
- By preserving relevant historical attention patterns through a memory-driven sparse attention matrix, the model can maintain a more consistent and accurate focus on visual information.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.