Reproduce “SynthCorp: A Platform for Creating Benchmarks for Data Agents” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jiani Zhang, Manav Garg, Eugenie Lai, Cheng Zhen, Eva Sharma, Yang Pan, Yaming Lin, Alon Halevy
SynthCorp generates customizable data-agent benchmarks by simulating business events that jointly create databases, documents, catalogs, questions, and executable reference programs. A hidden provenance log links artifacts over time and helps verify answers. The authors built seven estates with 256 tasks; all deterministic checks passed on six smaller estates. A Retail-1000 task exposed point-in-time failure, but does not show generalization to real companies.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.