Reproduce “SALUS: Automated Auditing of NL-to-SQL Benchmarks through Weak Supervision of Multi-Agent Output” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shiyuan Zhou, Ashwin Gerard Colaco, Sainyam Galhotra, Sharad Mehrotra
SALUS audits NL-to-SQL benchmark annotations by combining four SQL-generating agents’ execution, structural, and question-alignment signals. A generative label model converts their noisy votes into training labels, while a feature-conditioned model estimates which agents to trust for different queries. On BIRD-Clean-xs, SALUS detects erroneous references with 0.9194 F1; this transductive result is not evidence of transfer to new benchmarks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.