Reproduce “Normalization Layer Per-Example Gradients are Sufficient to Predict Gradient Noise Scale in Transformers” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Gavia Gray, Aman Tiwari, Shane Bergsma, Joel Hestness
Researchers from Cerebras Systems and subjective.dev developed an efficient method to compute per-example gradient norms, demonstrating that the Gradient Noise Scale (GNS) of normalization layers is sufficient to predict the total model GNS. This approach enables zero-overhead GNS tracking via custom kernels and reduced training time by 18% for a language model through GNS-guided batch size scheduling.
Key insights
- The proposed simultaneous per-example gradient norm computation offers lower FLOP costs compared to previous methods, particularly for linear layers with longer sequence lengths.
- The Gradient Noise Scale (GNS) of LayerNormalization layers is strongly correlated with the total model GNS and serves as a reliable, computationally efficient proxy for overall model GNS.
- Custom CUDA kernels can integrate per-example gradient norm computation for LayerNorm layers with zero throughput overhead, making GNS tracking practically free.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.