Reproduce “Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task” on OpenResearch

About this paper

Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li and 4 more

Spider is a large-scale, human-labeled dataset for complex and cross-domain text-to-SQL tasks, featuring 10,181 questions and 5,693 unique complex SQL queries across 200 databases. It is designed to evaluate generalization to new SQL structures and unseen database schemas, where state-of-the-art models achieve less than 13% exact matching accuracy.

Key insights

• Current state-of-the-art text-to-SQL models struggle significantly with Spider, particularly in generalizing to entirely new database schemas, as evidenced by a substantial performance drop in the Database Split setting.
• Models demonstrate the most difficulty with predicting the `WHERE` clause and accurately handling column prediction within SQL components, indicating challenges in understanding complex conditional logic and multi-column relationships.
• Sketch-based models, which incorporate SQL structure information, generally outperform Seq2Seq models by producing more valid SQL, but are limited in generating highly novel or very complex nested queries or set operations.

Summary by alphaXiv.