Reproduce “Information Geometry of Evolution of Neural Network Parameters While Training” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Abhiram Anand Thiruthummal, Eun-jin Kim, Sergiy Shelyag
This research investigates the training dynamics of Artificial Neural Networks by applying an information geometric framework to track the evolution of network parameters. It demonstrates that a specific information geometric measure, the second derivative of the logarithm of information length, consistently identifies phase transition-like behavior correlating with the onset of overfitting, without requiring a test dataset.
Key insights
- The collective behavior of ANN parameters undergoes measurable "phase transition-like" changes during training, detectable through information geometric analysis.
- A specific inflection point in the information geometric trajectory (a minimum in `d²(log L)/d(log t)²`) serves as an independent, internal indicator for the onset of overfitting.
- The observed information geometric transitions share mathematical analogies with thermodynamic phase transitions in statistical physics, including indications of finite-size scaling.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.