Reproduce “Luna: An Evaluation Foundation Model to Catch Language Model Hallucinations with High Accuracy and Low Cost” on OpenResearch

About this paper

Masha Belyi, Robert Friel, Shuai Shao, Atindriyo Sanyal

Luna is an evaluation foundation model designed to accurately detect hallucinations in Retrieval-Augmented Generation (RAG) systems with significantly reduced operational costs and latency. It effectively handles very long contexts, making it suitable for real-time industry applications.

Key insights

• Framing hallucination detection as a span-level NLI task and aggregating support probabilities across multiple context windows allows for robust performance even with very long input contexts.
• A moderately-sized encoder model, fine-tuned specifically for hallucination detection, can achieve accuracy comparable to much larger LLM-based evaluators while significantly reducing computational cost and latency.
• Luna's architecture demonstrates strong cross-domain generalization, accurately detecting hallucinations across diverse industry applications and RAG task types with minimal re-finetuning.

Summary by alphaXiv.