Reproduce “Weibull Weight-Scale Parameter Evolution under AdamW Training Dynamics” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tiexin Ding
Research provides a mechanistic explanation for the characteristic rise, overshoot, and relaxation trajectory of the Weibull weight-scale parameter λ during AdamW training by decomposing optimizer forces. It identifies the alignment force as dominant in the initial phase and links force balance to the observed weight-scale evolution.
Key insights
- The characteristic evolution of the Weibull weight-scale parameter λ (rise, overshoot, relaxation, and floor) is mechanistically explained by the dynamic balance of alignment, injection, and decay forces from the AdamW optimizer.
- The alignment force is the predominant driver of weight-scale growth in the early training phase, contributing 88-94% of the total absolute force budget for Transmission-class components.
- The established 'k-lock' property (Weibull shape parameter k ≈ 1.20) for Transmission-class layers provides a stable link between the weight's root-mean-square (RMS) and its Weibull scale parameter λ, validating the force decomposition's relevance to λ's behavior.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.