Reproduce “STLM Engineering Report: Dropout” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Dylan Hillier, Leon Guertler, Bobby Cheng, Cheston Tan
The authors test whether dropout helps small language models in overfitting and underfitting settings. Dropout randomly zeros activations during training, with schedules varying its rate. On Simple English Wikipedia, constant 0.1 dropout lowers in-domain perplexity from 2.26 to 1.88; in the larger-data setting, benefits are minor and inconsistent. These results do not test repeated-data training or establish an underfitting benefit.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.