Reproduce “FOCUS: Internal MLLM Representations for Efficient Fine-Grained Visual Question Answering” on OpenResearch

About this paper

Liangyu Zhong, Fabio Rosenthal, Joachim Sicking, Fabian Hüger, Thorsten Bagdonat, Hanno Gottschalk, Leo Schwinn

Researchers from Technical University of Berlin, Technical University of Munich, CARIAD SE, and Volkswagen AG developed FOCUS, a training-free visual cropping method that enhances multimodal large language model (MLLM) performance on fine-grained Visual Question Answering (VQA). This approach achieved up to 42% higher accuracy on LLaVA-1.5 and 17% higher on LLaVA-OneVision for fine-grained tasks, while requiring 3 to 6.5 times less computational effort than existing methods.

Key insights

• MLLMs' internal KV-cache value features can be effectively repurposed to construct object relevance maps, enabling precise localization without explicit attention weights or additional training.
• A confidence-based re-ranking mechanism for proposed ROIs significantly improves the robustness and accuracy of fine-grained detail identification by filtering out noisy relevance scores.
• Leveraging combined relevance maps and existence confidence allows for an efficient and training-free visual cropping approach that dramatically reduces computational overhead while boosting MLLM fine-grained perception.

Summary by alphaXiv.