Reproduce “Drop Dropout on Single-Epoch Language Model Pretraining” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Houjun Liu, John Bauer, Christopher D. Manning
Research from Stanford University systematically investigated the role of dropout in single-epoch language model pretraining, demonstrating that its removal consistently improves language modeling loss, linguistic understanding, and downstream task performance across various model scales and architectures. The study found that dropout is not only unnecessary but actively detrimental in this training regime, also showing differential impacts on knowledge editability.
Key insights
- Dropout is detrimental to language model pretraining in the single-epoch, massive-data regime, consistently degrading performance.
- The traditional benefits of dropout for regularization do not apply and are actively harmful under modern LLM training conditions.
- Dropout appears to interfere with optimal knowledge organization within models, affecting the success of certain knowledge editing techniques like MEND.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.