Reproduce “Datasets for Large Language Models: A Comprehensive Survey” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yang Liu, Jiahuan Cao, Chongyu Liu, Kai Ding, Lianwen Jin
This overview provides a comprehensive analysis of 444 diverse datasets crucial for Large Language Model (LLM) development, categorizing them by their role in the model's lifecycle. It details their characteristics, construction methods, and temporal trends across 8 language categories and 32 domains, identifying that pre-training corpora total 774.5 TB and other datasets contain over 700 million instances.
Key insights
- Webpages are the predominant data type in pre-training corpora, and there was a substantial increase in dataset creation in 2023.
- Instruction fine-tuning datasets are increasingly constructed using model-based methods, while human voting remains the most common approach for preference datasets.
- Code-based assessment is the most prevalent method for evaluation datasets, and English datasets continue to dominate across all types, highlighting a need for greater linguistic diversity.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.