Reproduce “Two Speeds of Learning: A Representation-Readout Decomposition of Grokking and Double Descent” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Chi-Ning Chou, Oscar Uzdelewicz, Neng-Chun Chiu, Yao-Yuan Yang, SueYeon Chung

Chou et al. introduced a representation-readout decomposition framework for deep neural networks that reveals two distinct speeds of learning underlying phenomena like grokking and epoch-wise double descent. This framework demonstrates that representation learning is a continuous process, while readout calibration can exhibit train-biased dynamics, and provides diagnostic signatures to differentiate genuine from spurious generalization.

Key insights

  • Representation learning is a continuous process, with significant refinement occurring before the abrupt emergence of test accuracy in grokking, contrary to a strict 'lazy-to-rich' transition.
  • The readout's learning direction is train-biased before grokking, indicated by a large train-test NTK alignment gap, signifying readout-level overfitting.
  • Different architectures (MLPs vs. Transformers) achieve generalization during grokking through qualitatively distinct geometric pathways in their representations, as revealed by GLUE measures.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.