Reproduce “Dynamical versus Bayesian Phase Transitions in a Toy Model of Superposition” on OpenResearch

About this paper

Zhongtian Chen, Edmund Lau, Jake Mendel, Susan Wei, Daniel Murfet

This work analyzes phase transitions in a Toy Model of Superposition (TMS), demonstrating a link between empirically observed dynamical loss drops during SGD training and theoretically predicted Bayesian phase transitions. It calculates local learning coefficients for different network configurations, showing that SGD implicitly seeks more complex solutions that correspond to lower loss and higher complexity phases.

Key insights

• Dynamical transitions observed during SGD training have corresponding Bayesian phase transitions that act as their "antecedents," implying a deep, non-coincidental relationship between the optimization trajectory and the structure of the Bayesian posterior.
• The local learning coefficient (λ) from Singular Learning Theory serves as a principled measure of model complexity, effectively quantifying degeneracy and correlating with both Bayesian and dynamical transitions.
• Learning in the Toy Model of Superposition (TMS) is characterized as a sequential journey through parameter space, moving from regions of high loss and low complexity to regions of low loss and high complexity.

Summary by alphaXiv.