Reproduce “A Guide to Misinformation Detection Data and Evaluation” on OpenResearch

About this paper

Camille Thibault, Jacob-Junqi Tian, Gabrielle Peloquin-Skulski, Taylor Lynn Curtis, James Zhou, Florence Laflamme, Yuxiang Guan, Reihaneh Rabbany and 2 more

A large-scale analysis of misinformation detection datasets revealed prevalent quality issues, including spurious correlations and low veracity assessment feasibility, while also demonstrating that standard metrics frequently mislabel valid generative AI outputs, with 38-55% of "incorrect" LLM predictions having sound rationales.

Key insights

• Many misinformation datasets exhibit significant spurious correlations (keyword or temporal) that enable models to achieve artificially high performance without learning intrinsic veracity.
• A substantial portion of claims in evaluated datasets is infeasible to assess for truthfulness without evidence retrieval, challenging the reliability of models trained on such ambiguous data.
• Traditional categorical evaluation metrics often under-evaluate generative AI systems, as a high percentage of LLM 'errors' are found to have valid rationales upon human inspection, indicating a mismatch between model output and rigid ground truth labels.

Summary by alphaXiv.