Reproduce “DataSpace: Benchmarking Data Agents for Verifiable Analytics over Heterogeneous Workspaces” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Boyan Li, Zhuowen Liang, Yupeng Xie, Xiaotian Lin, Tianqi Luo, Xinyu Liu, Yizhang Zhu, Zhangyang Peng and 6 more
DataSpace introduces a benchmark comprising 410 cross-language tasks over heterogeneous workspaces to evaluate data agents on verifiable analytics. The benchmark reveals that even frontier models struggle with cross-modality evidence integration and precise tabular result materialization, with the top-performing agent achieving an overall accuracy of 66.34%.
Key insights
- Current frontier multimodal models and agent architectures demonstrate significant performance degradation when tasks require integrating evidence across multiple modalities or involve complex relational join operations.
- A substantial portion of agent failures, particularly in top models like Grok 4.5 (52.2% of audited failures), originates from issues in precisely materializing the requested tabular output schema rather than core analytical missteps.
- The choice and design of the agent harness play a critical role, contributing to a 15.36-point accuracy spread among different harnesses using the same backbone model, highlighting the importance of system-level orchestration.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.