Reproduce “Semantic Evaluation for Text-to-SQL with Distilled Test Suites” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ruiqi Zhong, Tao Yu, Dan Klein
Researchers at UC Berkeley and Yale University introduce 'test suite accuracy,' a robust method for evaluating Text-to-SQL models by generating compact, schema-constrained database test suites through fuzzing. This approach empirically demonstrates zero false positives and significantly reduces false negatives compared to existing metrics like Exact Set Match (ESM), providing a more reliable assessment of semantic correctness.
Key insights
- Fuzzing with 'neighbor queries' allows for the distillation of small, yet highly effective database test suites that expose subtle semantic differences between SQL queries.
- Manual validation confirms the high precision of test suite accuracy, demonstrating it generates no false positives, unlike single denotation evaluation.
- Existing evaluation metrics like Exact Set Match (ESM) and single denotation accuracy have quantifiable false positive and false negative rates, particularly for complex queries, misrepresenting model performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.