Reproduce “SPARTA: Scalable and Principled Benchmark of Tree-Structured Multi-hop QA over Text and Tables” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Sungho Park, Jueun Kim, Wook-Shin Han
SPARTA introduces a scalable and principled framework for automatically generating large-scale Table–Text QA benchmarks by unifying textual and tabular data into a single relational store. It employs execution-guided LLMs to synthesize complex, tree-structured SQL queries with advanced analytical operations, which are then verbalized into natural language questions with lightweight human validation. State-of-the-art models exhibit substantial performance drops on this benchmark, revealing fundamental limitations in handling deep multi-hop reasoning, complex analytical operations, and cross-modal information fusion.
Key insights
- Combining execution-guided LLM generation with provenance-based refinement is crucial for cost-effectively producing semantically sound, executable complex SQL queries that mimic human-written inquiries.
- Unifying heterogeneous data (text and tables) into a single relational store is effective for generating questions that naturally blend information from multiple modalities without explicit cross-modal pointers.
- Current state-of-the-art Table–Text QA models exhibit fundamental weaknesses in handling deep multi-hop, tree-structured reasoning, complex analytical operations (like aggregation and grouping), and robust cross-modal information fusion.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.