Reproduce “Generating Benchmarks for Factuality Evaluation of Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Dor Muhlgay, Ori Ram, Inbal Magar, Yoav Levine, Nir Ratner, Yonatan Belinkov, Omri Abend, Kevin Leyton-Brown and 2 more
AI21 Labs developed FACTOR (Factual Assessment via Corpus TransfORmation), an automated framework that generates multi-choice benchmarks to evaluate language model factuality against a controlled set of facts in open-ended generation. The method provides a direct measure of factual accuracy, showing that retrieval augmentation consistently improves factuality and that FACTOR accuracy is a more reliable indicator of factual correctness than perplexity in open-ended text generation.
Key insights
- Factual knowledge in language models improves with increased model size, although even the largest models show substantial room for improvement in factual accuracy.
- Retrieval augmentation consistently enhances factual accuracy, with FACTOR providing the first automatic metric to quantify these improvements in a generation setting.
- FACTOR accuracy offers a distinct and more direct measure of factual grounding than perplexity, with human evaluations confirming it better predicts true claim generation in open-ended text.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.