Reproduce “Invent a Dataset: Measuring Dataset Generation Abilities with Zero Seed Data” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shivalika Singh, Andrija Djurisic, Gbemileke Onilude, Sudip Roy, Sara Hooker
Invent-a-Dataset turns a natural-language description into training-ready examples without requiring seed examples. The authors evaluate it on eight unconstrained dataset requests, measuring prompt and completion quality with an LLM judge and semantic diversity with DCScore, an embedding-based measure. At 5,000 samples, it scores 7.90/10 in quality and about 19% higher diversity than the next baseline, within this benchmark.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.