Reproduce “Retrieval-Augmented Multimodal Model for Fake News Detection” on OpenResearch

About this paper

Yiheng Li, Weihai Lu, Hanyi Yu, Yue Wang

A retrieval-augmented multimodal model (RAMM) enhances fake news detection by integrating external knowledge and narrative consistency via Multimodal Large Language Models (MLLMs), addressing challenges in cluster-based misinformation. The model achieved state-of-the-art F1-scores, including a 2.1% improvement on Weibo-21, by leveraging abstract narrative alignment and semantic representation alignment modules.

Key insights

• Explicitly integrating retrieval mechanisms with Multimodal Large Language Models (MLLMs) to capture abstract narrative consistency and external domain-specific knowledge is effective for detecting cluster-based fake news.
• The Common Information Bottleneck Loss (CIBL), through its alignment, reconstruction, and compression components, is vital for distilling robust and concise shared narrative representations, preventing issues like representation collapse.
• Leveraging analogical reasoning by selecting relevant demonstrations significantly improves MLLM semantic understanding and contributes to better generalization across diverse news domains.

Summary by alphaXiv.