Reproduce “KramaBench: A Benchmark for AI Systems on Data-to-Insight Pipelines over Data Lakes” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Eugenie Lai, Gerardo Vitagliano, Ziyu Zhang, Om Chabra, Sivaprasad Sudhir, Anna Zeng, Anton A. Zabreyko, Chenning Li and 11 more
KRAMABENCH evaluates whether AI systems can turn heterogeneous data lakes into reliable analytical answers, requiring file discovery, data cleaning, pipeline design, execution, and result formatting. Across 104 tasks in six scientific and applied domains, the best agent reaches 55.83% end-to-end accuracy, while humans score 76.75%; performance drops sharply on obscured data, exposing reliance on memorized information and weaknesses in grounded analysis.
Key insights
- Providing the correct files improves results but does not solve the core problem: the strongest oracle configuration reaches about 62.81%, showing that planning, interpretation, and execution remain major bottlenecks beyond retrieval.
- Models are better at describing high-level workflows than implementing data-dependent operations correctly; the best pipeline-design score is 41.71%, compared with 22.05% for the reported sub-task implementation setting.
- Obscured inputs cause large performance declines, in some configurations reducing accuracy from roughly 60% to about 13%, indicating that systems may exploit memorized associations with public datasets instead of grounding answers in supplied data.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.