Reproduce “Learning Rules, Memorizing Facts: An Information-Theoretic View of Hallucination” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Gengze Xu, Anxin Guo, Yifei Hu, Xiangyu Wu, Yong Liu, Ziqiao Wang
The authors present an information-theoretic framework that separates reusable rules, arbitrary facts, and training-sample selection. In a sparse random-fact setting, they show that bounded score-level information per fact imposes a lower bound on false positives at positive recall, even when all true facts are observed in training. Synthetic relational experiments show trade-offs, but do not measure information allocation in LLMs.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.