Reproduce “Position: The Most Expensive Part of an LLM should be its Training Data” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Nikhil Kandpal, Colin Raffel
Researchers from the University of Toronto introduce a novel economic analysis framework that estimates the true cost of LLM training data based on replacement value, revealing that data costs exceed traditional training costs by 10-1000x and represent an unsustainable fraction of most companies' annual revenue when fair compensation is considered.
Key insights
- Training dataset costs dwarf conventional training costs (300x higher for GPT-4)
- Fair data compensation would exceed 10% of annual revenue for 8 of 10 major LLM companies
- Dataset sizes double every 8 months, making uncompensated data collection increasingly unsustainable
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.