Reproduce “Dynamic Masking Rate Schedules for MLM Pretraining” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zachary Ankner, Naomi Saphra, Davis Blalock, Jonathan Frankle, Matthew L. Leavitt
Researchers from MosaicML, MIT, Harvard, and DatologyAI developed dynamic masking rate schedules for Masked Language Modeling pretraining, achieving Pareto improvement in training efficiency and downstream task performance for BERT-style models. The optimal linear decreasing schedule provided up to 1.89x speedup for BERT-base models and improved average GLUE accuracy by up to 0.46%.
Key insights
- A decreasing masking rate schedule (starting high and ending lower) is crucial for performance gains, supporting the idea that higher noise early in training aids optimization.
- The benefits of dynamic masking arise from both the increased information obfuscation (loss landscape smoothing) and the additional training signal from predicting more masked tokens.
- Dynamic scheduling improves fundamental linguistic understanding and generalizes to other pretraining objectives like Random Token Substitution (RTS).
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.