Reproduce “OAEI-LLM-T: A TBox Benchmark Dataset for Understanding Large Language Model Hallucinations in Ontology Matching” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhangcheng Qiang, Kerry Taylor, Weiqing Wang, Jing Jiang
OAEI-LLM-T, a new TBox benchmark dataset, provides a structured framework for identifying and categorizing Large Language Model (LLM) hallucinations in ontology matching (OM) tasks. The dataset supports benchmarking LLMs for OM by revealing their risk-aversion strategies and offers a method for generating fine-tuning data to reduce hallucination rates in LLM-based OM systems.
Key insights
- LLMs exhibit distinct 'risk-avoiding' (prioritizing precision, fewer proposed mappings) versus 'risk-seeking' (prioritizing recall, more proposed mappings) behaviors when performing ontology matching tasks, influencing their hallucination profiles.
- Incorrect mappings generated by LLMs in OM are often contextually relevant (e.g., aligning to a superclass or a related but disputed concept) rather than entirely false, highlighting challenges in semantic reasoning and contextual interpretation.
- The proposed OM-specific hallucination taxonomy allows for a granular understanding of LLM errors, enabling targeted model selection and fine-tuning strategies to mitigate specific types of hallucinations in structured knowledge domains.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.