Reproduce “The Stack: 3 TB of permissively licensed source code” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jia Li, Chenghao Mou, Carlos Muñoz Ferrandis, Yacine Jernite, Margaret Mitchell and 5 more
The Stack provides a comprehensive, publicly available dataset of 3.1 terabytes of permissively licensed source code from public GitHub repositories, designed for training Code LLMs. Models trained exclusively on this deduplicated dataset achieve competitive code generation performance on benchmarks like HumanEval and MBPP, demonstrating the viability of open and ethically sourced data for this task.
Key insights
- Near-deduplication of training data significantly improves Code LLM performance, showing up to 33% relative improvement on HumanEval pass@1.
- High-quality code generation can be achieved using exclusively permissively licensed data, yielding competitive results with existing state-of-the-art models.
- The Stack dataset offers superior quality and comprehensive coverage compared to alternative public datasets, leading to better model performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.