Reproduce “Measuring Structured Predictability in Neural Training Dynamics: A Cross-Regime Study” on OpenResearch

About this paper

Fanqi Wang, Weisheng Tang, Hairong Qi

This research introduces a multiscale framework to measure the short-horizon predictability of neural network parameter updates during training, revealing how this temporal structure varies across parameter types and training conditions. It found that auxiliary parameters exhibit higher predictability than bulk parameters, which nonetheless contain transient "predictable pockets," and that predictability is systematically modulated by architecture, optimizer, and weight decay.

Key insights

• The developed probes consistently track intrinsic trajectory structure and correlate with realized optimization states, validating their utility as diagnostics.
• Short-horizon predictability is spatially localized and dynamically evolving, with auxiliary parameters showing simpler dynamics and bulk parameters containing transient, shifting "predictable pockets."
• Training conditions such as architecture, optimizer family, and regularization systematically modulate the level and localization of predictability in parameter updates.

Summary by alphaXiv.