Reproduce “When Ground Truth Is Not Ground Truth: Automatic Labeling for Hallucination Detection” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Rounak Sen, Divya Kumari, Deeksha Varshney
The authors compare automatic judges that label generated answers as truthful or hallucinated with human annotations, testing their use as supervision for hallucination detectors. They find detector scores depend on the evaluation judge: cross-judge transfer averages 0.530 AUROC, versus 0.669 when training and evaluation use the same judge. QGen-9B’s labels misclassify 12.86% against humans, within these held-out datasets.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.