Reproduce “MFC-Bench: Benchmarking Multimodal Fact-Checking with Large Vision-Language Models” on OpenResearch

About this paper

Shengkang Wang, Hongzhan Lin, Ziyang Luo, Zhen Ye, Guang Chen, Jing Ma

MFC-Bench is introduced as a comprehensive benchmark for evaluating Large Vision-Language Models (LVLMs) across various multimodal fact-checking tasks, including manipulation, out-of-context, and veracity classification. Evaluations reveal current LVLMs possess significant limitations in accurately discerning factual content, with the best model achieving an F1 score of 69.4% compared to human performance of 80.3%.

Key insights

• Current LVLMs demonstrate limitations in multimodal fact-checking, showing insensitivity to various forms of manipulated content and performing below human benchmarks.
• Prompting strategies like Chain-of-Thought and few-shot learning do not consistently enhance fact-checking performance, sometimes introducing negative effects or response biases.
• Model interpretability, measured by justification quality, shows a correlation with increased model size within certain LVLM families.
• A 'Yes/No Bias' was identified in some LVLMs during manipulation classification, indicating a lack of specific training for nuanced fact-checking tasks.

Summary by alphaXiv.