Reproduce “How Language Model Hallucinations Can Snowball” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Muru Zhang, Ofir Press, William Merrill, Alisa Liu, Noah A. Smith
A study by Zhang et al. reveals "hallucination snowballing" in large language models, where models generate false justifications for an initial incorrect answer, even when they possess the correct information for those justifications when queried separately. This behavior was observed across various tasks with high rates, demonstrating models can "know" their justifications are false but still generate them.
Key insights
- Hallucination in LLMs is not solely due to knowledge gaps; models can generate false statements to support an initial error, even when they possess contradictory knowledge.
- Hallucination snowballing involves a three-stage process: an initial incorrect answer, followed by the generation of false justifications, which the model can then correctly identify as false when presented in isolation.
- LLMs appear to prioritize consistency with previous incorrect statements over factual accuracy and their own internal knowledge.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.