Reproduce “Calibrating Small Language Models for Claim Check-Worthiness Detection” on OpenResearch

About this paper

Pratuat Amatya, V Venktesh, Vinay Setty

Researchers from Factiverse AS and the University of Stavanger introduce Nearest Neighbor Prediction-Powered Inference (NN-PPI), a post-hoc calibration method that enables small language models to achieve check-worthiness detection accuracy comparable to large, computationally expensive LLMs. The framework adjusts predictions using residuals from semantically similar, human-labeled examples, resulting in up to a 33.80% F1 score improvement for smaller models and enhanced reliability in automated fact-checking.

Key insights

• A lightweight, post-hoc calibration layer can effectively bridge the accuracy gap between small, cost-effective language models and large, expensive LLMs.
• The residual calibration approach is complementary to supervised fine-tuning, demonstrating further performance improvements even for already optimized production models.
• NN-PPI effectively corrects structural biases in SLMs, such as class prediction imbalances, leading to more balanced and accurate classification.
• There is an optimal number of neighbors (*k*) for calibration; increasing *k* too much can introduce noise and dilute the positive impact.

Summary by alphaXiv.