Reproduce “Layers, Sinks, and Scaling: Adaptive Evidence Selection for Multimodal Large Language Models” on OpenResearch

About this paper

Zhenbin Wang, Lei Zhang, Lituan Wang, Wei Huang, Yan Wang, Zhenwei Zhang

AREA is a training-free inference framework that adaptively controls how frozen multimodal language models consume retrieved text and visual evidence. By gating unnecessary interventions, scaling evidence to its estimated spread, and refreshing text selection during decoding, it improves average KB-VQA performance by roughly 1.9–5.9 points across nine MLLM checkpoints and outperforms Look Twice by about 0.6 points, while also reducing reported runtime by 7–34%.

Key insights

• Evidence control is decomposed into three independent decisions: whether to intervene, how much evidence to expose, and whether textual evidence must be updated as generation proceeds.
• Visual attention sinks can be treated differently for localization and gating: filtered relevance identifies candidate regions, while raw sink concentration indicates when the visual signal may be unreliable.
• Adaptive evidence presentation is most useful when answers require a compact link between visual entity recognition and external facts; it is less suitable for tasks that depend on global image structure, such as some chart questions.

Summary by alphaXiv.