Reproduce “A Multi-agent Text2SQL Framework using Small Language Models and Execution Feedback” on OpenResearch

About this paper

Thanh Dat Hoang, Thanh Trung Huynh, Matthias Weidlich, Thanh Tam Nguyen, Tong Chen, Hongzhi Yin, Quoc Viet Hung Nguyen

MATS, a multi-agent framework, leverages small language models (SLMs) and execution feedback to translate natural language into SQL queries, achieving execution accuracy comparable to large language models while operating on significantly fewer computational resources (e.g., 21.2 GB GPU memory). This approach enhances data privacy and reduces costs by enabling in-house deployment.

Key insights

• Decomposing complex Text2SQL tasks into specialized sub-tasks allows SLMs to overcome their individual reasoning limitations and collectively achieve high performance.
• Integrating database execution feedback through RLEF provides an effective and scalable mechanism for agent alignment, enabling SLMs to learn correct SQL generation without explicit human labels.
• The modified ORPO algorithm proves superior for SLM alignment, effectively balancing supervised fine-tuning gains with preference learning while consuming less memory and training time compared to PPO and DPO.

Summary by alphaXiv.