Reproduce “Latent State Models of Training Dynamics” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Michael Y. Hu, Angelica Chen, Naomi Saphra, Kyunghyun Cho
Researchers at NYU utilized Hidden Markov Models to analyze the internal training dynamics of neural networks, demonstrating how random seeds lead to varying 'training maps' with identifiable 'detour states' that predict convergence efficiency. Their framework reveals that training variability, including the grokking phenomenon, is largely modulated by architectural and optimization choices affecting the loss landscape.
Key insights
- HMMs effectively create low-dimensional, predictive 'training maps' that summarize complex neural network training dynamics and highlight key metric changes during state transitions.
- Training variability and phenomena like grokking are not inherent properties of a task but are largely modulated by architectural and optimization choices that affect the smoothness and stability of the loss landscape.
- 'Detour states,' identified through the HMM framework, are optional latent states associated with increased convergence time, serving as direct indicators of training instability caused by random initialization.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.