Reproduce “Unsupervised Pretraining for Fact Verification by Language Model Distillation” on OpenResearch

About this paper

Adrián Bazaga, Pietro Liò, Gos Micklem

SFAVEL, an unsupervised pretraining framework for fact verification, distills high-quality claim-fact alignments from frozen language models into a knowledge model. The framework achieves state-of-the-art performance on the FEVER benchmark with an 86.15% FEVER score and competitive results on FB15k-237, surpassing both previous unsupervised and supervised methods without relying on human-annotated data.

Key insights

• Unsupervised pretraining using language model distillation can achieve state-of-the-art performance in fact verification by effectively learning high-quality claim-fact alignments.
• A composite loss function that integrates feature distillation, intra-sample contrastive learning, and a scoring objective is crucial for transferring semantic knowledge and discriminating between relevant and irrelevant evidence.
• The size and quality of the pre-trained language model backbone directly influence the distilled model's performance, indicating that richer source features lead to more effective unsupervised representations.

Summary by alphaXiv.