Reproduce “The Hallucinations Leaderboard -- An Open Effort to Measure Hallucinations in Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Giwon Hong, Aryo Pradipta Gema, Rohit Saxena, Xiaotang Du, Ping Nie, Yu Zhao, Laura Perez-Beltrachini, Max Ryabinin and 3 more
The Hallucinations Leaderboard provides an open, standardized platform to measure and compare LLM hallucination tendencies across diverse tasks, distinguishing factuality from faithfulness. It finds that instruction fine-tuning improves faithfulness but has mixed effects on factuality, while increased model size generally boosts both, particularly factuality.
Key insights
- Hallucination tendency in LLMs is more strongly influenced by model family and shared pre-training data than by instruction fine-tuning.
- Instruction fine-tuning generally improves faithfulness (adherence to context/instructions) but shows mixed or even declining effects on factuality (adherence to world knowledge), suggesting a potential trade-off.
- Increasing model size generally improves both faithfulness and factuality, with factuality scores exhibiting more substantial gains, indicating that scaling primarily enhances the model's ability to retain and utilize factual knowledge.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.