Reproduce “The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AI” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shayne Longpre, Robert Mahari, Anthony Chen, Naana Obeng-Marnu, Damien Sileo, William Brannon, Niklas Muennighoff, Nathan Khazam and 9 more
The Data Provenance Initiative conducted a large-scale, multi-institutional audit of over 1800 text datasets used for AI finetuning, revealing pervasive license omissions and mischaracterizations on major platforms. This work introduces the Data Provenance Explorer and Data Provenance Cards to provide detailed, legally reviewed metadata, empirically demonstrating a significant divide in diversity and availability between commercially viable and non-commercial datasets.
Key insights
- The AI data ecosystem suffers from pervasive license opacity, with major hosting platforms exhibiting high rates of missing and mischaracterized licenses, significantly understating legal restrictions.
- There is a sharp and widening divide between commercially viable and non-commercial/academic-only datasets, with non-commercial data demonstrating statistically greater diversity in tasks, topics, and source types.
- A substantial portion of newer and synthetic datasets, particularly those generated using commercial APIs, are predominantly licensed for non-commercial use, limiting their application in commercial AI development.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.