Reproduce “Why do small language models underperform? Studying Language Model Saturation via the Softmax Bottleneck” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Nathan Godey, Éric de la Clergerie, Benoît Sagot
Small language models underperform and saturate during training due to a 'softmax bottleneck' and representation degeneration. Research from Inria Paris and Sorbonne Université identified that the linear prediction head requires approximately 1000 hidden dimensions to avoid significant performance drops, explaining the observed limitations.
Key insights
- Performance saturation in small LMs strongly correlates with the emergence of high last-layer anisotropy and a collapse in the effective rank of the linear language modeling head.
- The linear LM prediction head acts as a 'softmax bottleneck,' requiring a hidden dimension of approximately 1000 to avoid significant performance degradation.
- Natural language's inherent contextual probability distribution has a very high dimensionality (10,000-15,000 for negligible error), posing a significant challenge for low-rank linear heads.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.