Reproduce “A Benchmark to Understand the Role of Knowledge Graphs on Large Language Model's Accuracy for Question Answering on Enterprise SQL Databases” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Juan Sequeda, Dean Allemang, Bryon Jacob
This research from data.world quantifies the impact of Knowledge Graphs on the accuracy of Large Language Models answering natural language questions over enterprise SQL databases. The study finds that integrating a Knowledge Graph improves overall execution accuracy by 37.5%, from 16.7% to 54.2%, and enables answers for complex queries where direct interaction with SQL schemas fails.
Key insights
- Providing a structured business context via a Knowledge Graph is essential for improving LLM accuracy in enterprise question answering scenarios.
- Direct LLM interaction with raw SQL DDL is insufficient for complex enterprise schemas, often resulting in complete failure for high-schema-complexity questions.
- Inaccuracies shifted from severe "hallucinations" in direct SQL to more manageable "path inconsistencies" when using Knowledge Graphs and SPARQL.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.