Reproduce “FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer and 1 more
FActScore evaluates long-form factuality by splitting answers into atomic facts and measuring the share supported by a chosen knowledge source. It uses retrieval and an evaluator language model to approximate human checks at scale. In biographies checked against English Wikipedia, ChatGPT’s facts scored 58.3% supported; this measures precision for that setup, not completeness or all ChatGPT answers.
Key insights
- Factual precision in long-form text generation from Large Language Models (LLMs) significantly declines for less common entities and for facts appearing later in the generated text.
- Integrating retrieval mechanisms is crucial for improving the accuracy of automated factual evaluation, and ensembling different validation approaches can further reduce estimation error.
- Despite advancements, current state-of-the-art LLMs, including leading commercial models, are considerably less factual in long-form generation than human-written content.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.