Reproduce “Metasql: A Generate-then-Rank Framework for Natural Language to SQL Translation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yuankai Fan, Zhenying He, Tonghui Ren, Can Huang, Yinan Jing, Kai Zhang, X.Sean Wang
Fudan University researchers developed METASQL, a meta-framework that enhances existing Natural Language to SQL (NL2SQL) translation models through a unified generate-then-rank approach. The framework achieved a 15.3% improvement in GPT-4's Exact Match accuracy and a 21.5% improvement in LGESQL's Execution Match accuracy on the SPIDER benchmark, particularly on complex queries.
Key insights
- Separating the generation of candidate SQL queries from their selection allows for greater diversity and global optimization in NL2SQL systems.
- Using extracted high-level semantic metadata as control signals during generation can effectively guide models to produce a wider range of relevant SQL candidates.
- A multi-stage ranking approach, moving from coarse-grained sentence-level matching to fine-grained phrase-level matching, is effective for precise SQL query selection.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.