Reproduce “Why Adaptive Optimizers Underestimate Rare Tokens” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Sangsidhya Kar
The paper shows that coordinate-wise adaptive optimizers can underpredict rare softmax tokens, not merely learn them slowly. It attributes this to second-moment normalization: after occurrence, a rare token’s upward correction is suppressed, while later downward updates are less suppressed. In a small synthetic language model, Adam at β₂=0.95 gives mean r_y=-2.72 versus AMSGrad’s -0.14; effects in large models remain open.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.