Reproduce “Hallucination Detection in Large Language Models with Metamorphic Relations” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Borui Yang, Md Afif Al Mamun, Jie M. Zhang, Gias Uddin
MetaQA, developed by researchers including those from King’s College London, presents a self-contained, zero-resource framework for detecting fact-conflicting hallucinations in Large Language Models. This method utilizes metamorphic relations to generate and verify diverse mutations of LLM responses, consistently outperforming SelfCheckGPT with an average F1-score improvement of up to 112.2% across multiple LLMs and datasets.
Key insights
- Applying synonymy and antonymy-based Metamorphic Relations to LLM responses effectively exposes factual inconsistencies (hallucinations) that simpler consistency checks might miss.
- Using the LLM under test as its own 'test oracle' for both generating and verifying mutations creates a robust, self-contained, and model-agnostic hallucination detection system.
- Generating diverse semantically related and opposed mutations provides a more comprehensive and effective assessment of factual accuracy compared to merely checking for consistency among identical re-generations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.