Reproduce “Sub-Scaling Laws: On the Role of Data Density and Training Strategies in LLMs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhengyu Chen, Siqi Wang, Teng Xiao, Yudong Wang, Shiqi Chen, Xunliang Cai, Junxian He, Jingang Wang
This research systematically investigates the phenomenon of sub-scaling in large language models, where performance gains decelerate despite increased training resources. The study identifies high data density and non-optimal training resource allocation as key drivers of this behavior, and it proposes a sub-optimal scaling law that more accurately predicts performance in these regimes.
Key insights
- Sub-scaling in large language models is primarily driven by high data density, which leads to redundant information, and non-optimal resource allocation, particularly over-training.
- The effectiveness of additional data diminishes with increasing data density due to a decrease in novel information gain, affecting performance scaling.
- A critical Over-Training Ratio (OTR) threshold of around 50 (training tokens/model parameters) was identified, beyond which increasing the D/N ratio yields significantly diminishing returns on performance improvement.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.