Reproduce “BI-Agent and BI-Bench: Towards Automating End-to-End Business Intelligence” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Chuxuan Hu, Yeye He, Penny Zhou, Wee Hyong Tok, Daniel Kang, Surajit Chaudhuri

BI-Bench evaluates business-intelligence agents on the full path from raw, heterogeneous tables to answer tables, rather than isolating SQL generation. BI-Agent combines an LLM orchestrator with tools for table search, reshaping, join discovery, and code execution; specialized tools raise average accuracy by about 14 percentage points for SQL and 11 for Python, while post-training improves an 8B model from 5.2% to 35.0% with tools.

Key insights

  • End-to-end BI is considerably harder than conventional NL2SQL: the strongest vanilla model reaches 48.2% SQL accuracy, while leading NL2SQL systems score roughly 6.0%–26.3% on BI-Bench.
  • Delegating structural data operations to specialized tools complements, rather than replaces, LLM reasoning; tool assistance reduces transformation and table-selection errors, interaction rounds, and output length.
  • Post-training can make a small model more capable and economical: Qwen3-8B-RL with tools reaches 35.0% accuracy and costs about $0.19 per benchmark execution under the reported assumptions.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.