Reproduce “SQLens: An End-to-End Framework for Error Detection and Correction in Text-to-SQL” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yue Gong, Chuan Lei, Xiao Qin, Kapil Vaidya, Balakrishnan Narayanaswamy, Tim Kraska

SQLENS, an end-to-end framework from AWS, The University of Chicago, and MIT, addresses the issue of semantically incorrect SQL queries generated by Large Language Models (LLMs) by integrating diverse database and LLM-based error signals for fine-grained detection and iterative correction. The framework boosts Text-to-SQL system execution accuracy on benchmarks like BIRD by up to 20.50% and achieves an F1 score of 78.88 for error detection.

Key insights

  • Combining diverse error signals from both the database (which are efficient and high precision) and the LLM (which captures semantic nuances) is more effective for semantic error detection than relying solely on LLM self-reflection.
  • Weak supervision can effectively aggregate noisy, disparate error signals to create robust error detection models without requiring extensive ground-truth labels, which is a practical advantage in real-world deployment.
  • An iterative, fine-grained correction process, guided by prioritized error reports and guardrails, reduces the risk of introducing new errors compared to attempting to fix all detected issues simultaneously.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.