Reproduce “Grokking modular arithmetic” on OpenResearch

About this paper

Andrey Gromov

This research provides a fully interpretable model of the "grokking" phenomenon, showing that deep neural networks can achieve delayed generalization on modular arithmetic tasks without explicit regularization. The study analytically derives and empirically validates that grokking corresponds to networks learning periodic feature maps through a mechanism of constructive and destructive interference of trigonometric waves.

Key insights

• Grokking is an intrinsic property of the learning dynamics in specific settings and can occur with vanilla gradient descent, without the necessity of explicit regularization.
• The network learns specific periodic feature maps during grokking, which are characterized by Fourier components directly determined by the modulus of the arithmetic task.
• The onset of grokking coincides with a sharp increase in the growth rate of the Inverse Participation Ratio (IPR), quantifying the formation and localization of periodic features in Fourier space.

Summary by alphaXiv.