Reproduce “LettuceDetect: A Hallucination Detection Framework for RAG Applications” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ádám Kovács, Gábor Recski
LettuceDetect is a framework designed to identify token-level hallucinations in Retrieval-Augmented Generation (RAG) systems by leveraging a ModernBERT backbone for token classification. The system achieves an F1 score of 79.22% for example-level hallucination detection and 58.93% for span-level detection on the RAGTruth benchmark, processing 30-60 examples per second.
Key insights
- Framing hallucination detection as a token classification task enables precise identification of unsupported content at a granular level.
- Utilizing ModernBERT's long-context capabilities allows for processing significantly larger external knowledge documents (up to 8,192 tokens) than previous encoder-based methods.
- The system's efficiency allows for real-time hallucination detection and feedback for RAG pipeline improvement.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.