Reproduce “Temporally Consistent Factuality Probing for Large Language Models” on OpenResearch

About this paper

Ashutosh Bajpai, Aaryan Goyal, Atif Anwer, Tanmoy Chakraborty

IIT Delhi researchers defined the Temporally Consistent Factuality Probe (TeCFaP) task and a new dataset, TEMP-COFAC, to assess Large Language Models' (LLMs) ability to provide consistent and factually accurate responses to time-sensitive queries. They found existing LLMs exhibited low performance (0-1.48% for temporally consistent factuality) and proposed CoTSeLF, a fine-tuning framework that improved this metric by 90.4% compared to a strong baseline.

Key insights

• Current state-of-the-art LLMs, including open-source and commercial models, perform poorly on temporally consistent factuality, achieving between 0% and 1.48% in zero-shot settings.
• LLMs show better temporal factuality for historical entities (1500-1800) but better temporally consistent factuality for contemporary ones (after 1800), indicating different impacts of pre-training data.
• Optimizing LLMs by jointly considering consistency and temporal sensitivity through a reinforcement learning framework, as in CoTSeLF, provides substantial performance improvements.

Summary by alphaXiv.