Reproduce “TrustScore: Reference-Free Evaluation of LLM Response Trustworthiness” on OpenResearch

About this paper

Danna Zheng, Danyang Liu, Mirella Lapata, Jeff Z. Pan

A framework named TrustScore, developed by researchers at the University of Edinburgh, provides a reference-free method to evaluate the trustworthiness of large language model responses in closed-book question-answering. This approach utilizes an iterative multi-choice questioning technique to measure behavioral consistency, which, when integrated with external fact-checking, yields a trustworthiness score that correlates more strongly with human judgments than prior reference-free or many reference-based metrics.

Key insights

• "Behavioral Consistency," objectively measured through iterative querying against diverse distractors, effectively probes an LLM's intrinsic knowledge and serves as a robust indicator of trustworthiness.
• A sophisticated, priority-based algorithm for generating distractors, leveraging entities and embedding similarities, is critical for rigorous and challenging consistency checks.
• Combining internal behavioral consistency with external fact-checking provides a more accurate and holistic trustworthiness assessment, surpassing standalone fact-checking or many reference-based methods.

Summary by alphaXiv.