Reproduce “Weakly Supervised Veracity Classification with LLM-Predicted Credibility Signals” on OpenResearch

About this paper

João A. Leite, Olesya Razuvayevskaya, Kalina Bontcheva, Carolina Scarton

Researchers at The University of Sheffield developed Pastel, a weakly supervised framework that uses Large Language Models to extract expert-defined credibility signals from text for automatic veracity classification. This method achieves performance comparable to supervised state-of-the-art models in-domain and demonstrates superior robustness in cross-domain settings, without requiring direct human annotation for training.

Key insights

• LLMs can effectively extract complex, human-interpretable credibility signals in a zero-shot manner, bridging the gap between expert heuristics and automated systems.
• Weak supervision, specifically Snorkel's label model, can robustly combine multiple noisy LLM-extracted signals into accurate weak labels for veracity classification.
• The collective aggregation of diverse credibility signals is crucial for robust performance, with no single signal being overwhelmingly dominant.

Summary by alphaXiv.