Reproduce “Semantic Layers for Reliable LLM-Powered Data Analytics: A Paired Benchmark of Accuracy and Hallucination Across Three Frontier Models” on OpenResearch

About this paper

Michael Rumiantsau, Ivan Fokeev

This research provides empirical evidence that providing a semantic layer document to Large Language Models (LLMs) significantly enhances their analytical accuracy and reduces hallucinations when querying databases with natural language. Three frontier LLMs demonstrated an average increase of 17 to 23 percentage points in first-shot analytical accuracy on a realistic retail dataset when augmented with a 4 KB semantic layer.

Key insights

• Semantic layers directly address dominant text-to-SQL error categories, such as schema linking and business-logic misinterpretation, by providing explicit mappings and rules.
• The observed accuracy gains are a structural effect of how the task is framed to the LLM, rather than solely an improvement in the LLM's inherent generative capabilities.
• Explicitly articulating implicit business knowledge in a semantic layer prevents errors that LLMs cannot reliably infer from Data Definition Language (DDL) alone.

Summary by alphaXiv.