Reproduce “STLM Engineering Report: Dropout” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Dylan Hillier, Leon Guertler, Bobby Cheng, Cheston Tan

The authors test whether dropout helps small language models in overfitting and underfitting settings. Dropout randomly zeros activations during training, with schedules varying its rate. On Simple English Wikipedia, constant 0.1 dropout lowers in-domain perplexity from 2.26 to 1.88; in the larger-data setting, benefits are minor and inconsistent. These results do not test repeated-data training or establish an underfitting benefit.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.