Reproduce “A Scaling Theory of Multi-Epoch Pretraining: Late-Time Overfitting and Ensembling” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhen Yang, Yidi Miao, Blake Bordelon
The authors study allocating training passes and independently initialized models when language models reuse a fixed corpus. Averaging model logits reduces late-rising validation loss by suppressing prediction variance across models. On 100-million-token FineWeb, four base Transformers achieve validation cross-entropy 0.16 nats per token below one twice-wide model under a matched parameter-token budget; this comparison uses a common training horizon, and the theory remains qualitative for Transformers.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.