Reproduce “Bridging Lottery Ticket and Grokking: Understanding Grokking from Inner Structure of Networks” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Gouki Minegishi, Yusuke Iwasawa, Yutaka Matsuo

The University of Tokyo researchers demonstrated that the grokking phenomenon, characterized by delayed generalization, is driven by the emergence of "good subnetworks" or "lottery tickets" within a neural network's structure. These "grokked tickets," extracted after generalization, significantly accelerate the learning process, achieving generalization almost concurrently with memorization due to their specific structural configurations.

Key insights

  • The core mechanism behind grokking is the discovery of a 'good structural configuration' (a sparse subnetwork), rather than solely reduced weight norms or general network sparsity.
  • Significant structural changes within the network, quantifiable by metrics like Jaccard distance, are tightly synchronized with improvements in generalization performance.
  • Weight decay, commonly viewed as a direct regularizer, functions as an indirect mechanism for exploring and discovering beneficial network structures during training.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.