Reproduce “Data Predictability Shapes Weibull Weight-Scale Growth in Transformer Training” on OpenResearch

About this paper

Tiexin Ding

A quantitative relationship between a Transformer model's Weibull weight-scale parameter growth and the bigram conditional entropy of its training data is established. The research formulates a learning-rate-conditioned law that accurately predicts this growth within corrupted text corpora and identifies data redundancy as a factor limiting cross-corpus generalizability.

Key insights

• The convex functional form of the law, (λ² - λ₀²) = C₀(η) + C₁(η) (H_r - D)^0.59, originates from the data-side saturation of bigram conditional entropy as local structure is lost, not directly from the training dynamics.
• The learning rate (η) conditions weight-scale growth through power-law coefficients C₀(η) and C₁(η), representing an injection floor and alignment efficiency, which enables runs across different learning rates to collapse onto a shared functional form.
• The Weibull (k, λ) framework serves as a sensitive diagnostic ruler at both global and per-layer granularities, revealing that feed-forward blocks, particularly in deeper layers, exhibit higher data sensitivity to predictability.

Summary by alphaXiv.