Reproduce “Improving Automatic VQA Evaluation Using Large Language Models” on OpenResearch

About this paper

Oscar Mañas, Benno Krojer, Aishwarya Agrawal

This paper introduces LAVE (LLM-Assisted VQA Evaluation), a novel framework that leverages instruction-tuned Large Language Models to evaluate Visual Question Answering (VQA) model outputs. LAVE significantly outperforms traditional VQA Accuracy and other baseline metrics, achieving a Spearman correlation of 68.91 with human judgments when using GPT-3.5-Turbo, providing a more accurate assessment of generative VQA models.

Key insights

• Instruction-tuned LLMs, especially with few-shot prompting, demonstrate a strong capability to align with human judgments in evaluating VQA answers, overcoming the limitations of exact-match and n-gram-based metrics.
• The explicit generation of a rationale by the LLM before rating significantly improves evaluation performance, particularly for complex VQA scenarios with multiple reference answers.
• LAVE effectively addresses common failure modes of VQA Accuracy, such as handling synonyms and varying levels of answer specificity, by performing nuanced semantic understanding.

Summary by alphaXiv.