Reproduce “VeriTaS: The First Dynamic Benchmark for Multimodal Automated Fact-Checking” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Mark Rothermel, Marcus Kornmann, Marcus Rohrbach, Anna Rohrbach
VERITAS, developed by the Multimodal AI Lab at the Technical University of Darmstadt, introduces a dynamic and multimodal automated fact-checking benchmark designed to counteract data leakage by continuously incorporating new, real-world claims. It features fine-grained, uncertainty-aware annotations across five distinct properties, which were validated against human judgments with a 97.5% 3-bin accuracy, and demonstrates that current state-of-the-art AFC models still have substantial performance limitations, particularly on post-knowledge cutoff data and video content.
Key insights
- The research provides empirical evidence of data leakage, demonstrating a marked increase in Mean Squared Error (MSE) for all evaluated models after their knowledge cutoff dates on the VERITAS longitudinal split.
- Multimodal Automated Fact-Checking remains an unsolved problem, with even state-of-the-art models showing substantial performance gaps; the best model achieved an MSE of 0.183 on the most recent Q1 2026 data.
- Video claims pose the most significant challenge for almost all current AFC models, indicating a critical area for future improvement and research in multimodal understanding.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.