Reproduce “Grokking Beyond Neural Networks: An Empirical Exploration with Model Complexity” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jack Miller, Charles O'Neill, Thang Bui
Researchers at ANU empirically demonstrated that the grokking phenomenon, previously thought exclusive to neural networks, also occurs in Gaussian Process models and linear regression. They propose a general mechanism where grokking arises from models transitioning from readily accessible low-error, high-complexity solutions to harder-to-find low-error, low-complexity solutions, driven by regularization.
Key insights
- Grokking is a more fundamental phenomenon, not confined to neural networks, indicating its roots in general learning dynamics rather than specific architectural properties.
- Explicit complexity regularization (e.g., weight decay, KL divergence) is critical for driving models from readily accessible memorized (high-complexity) solutions to generalizable (low-complexity) solutions, which manifests as grokking.
- The 'grokking gap' can be reliably controlled and amplified by increasing the amount of 'concealed' or spurious information in the input data, suggesting a link between input dimensionality and generalization difficulty.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.