Reproduce “Increasing the LLM Accuracy for Question Answering: Ontologies to the Rescue!” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Dean Allemang, Juan Sequeda
Researchers from data.world AI Lab improved the accuracy of LLM-based question answering over enterprise structured data by integrating ontology-based query checking and LLM-powered repair mechanisms. Their approach increased accuracy from 54.2% to 72.6% while reducing the rate of incorrect answers to 19.4%.
Key insights
- A hybrid approach combining symbolic knowledge (ontologies) with LLM generative capabilities yields higher accuracy than either method in isolation for data access.
- Ontologies provide crucial semantic context and constraints that help identify and correct common LLM errors in query generation, especially regarding domain and range property usage.
- High-quality metadata and knowledge graph representations are essential for building robust and reliable LLM-based question answering systems over structured enterprise data.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.