Reproduce “LOVA3: Learning to Visual Question Answering, Asking and Assessment” on OpenResearch

About this paper

Henry Hengyuan Zhao, Pan Zhou, Difei Gao, Zechen Bai, Mike Zheng Shou

LOVA^3, developed by Show Lab at NUS, enhances Multimodal Large Language Models by integrating training for question asking and answer assessment, moving beyond traditional visual question answering. The framework improves performance across various VQA and multimodal benchmarks, with LOVA^3-7B achieving a +1.8% gain on VQAv2 and a +42.0% improvement on MME compared to LLaVA-1.5, and introduces the EvalQABench dataset for assessing answer correctness.

Key insights

• Explicitly training Multimodal Large Language Models (MLLMs) on question generation and answer assessment capabilities leads to a more comprehensive and robust multimodal understanding, surpassing performance on traditional Visual Question Answering (VQA) benchmarks.
• Even a relatively small, carefully curated dataset for answer assessment (EvalQABench) can significantly enhance an MLLM's evaluative reasoning and overall performance.
• A pipeline for creating high-quality synthetic data for complex tasks like answer assessment is robust and cost-effective when leveraging open-source MLLMs with human-in-the-loop refinement.

Summary by alphaXiv.