Reproduce “Algebraic Representability as the Limiting Regime of Grokking: An Exactly Solvable Model with Holomorphic Activations” on OpenResearch

About this paper

Chon-Fai Kam, Xavier Cadet, Miloud Bessafi, Frederic Cadet

Research by Kam et al. demonstrates that architectural expressivity can fundamentally limit neural network learning, presenting a "limiting regime" of grokking where memorization itself is constrained. Using an exactly solvable complex-valued network, the work shows that learning outcomes are binary (instant generalization or outright failure) based on a target function's algebraic representability, a stark contrast to the continuous spectrum of behaviors in universal approximators.

Key insights

• The holomorphic network's expressible function class is fixed by its activation degree $k$ and is exactly characterized as a $(k+1)$-dimensional subspace in the Fourier domain, independent of hidden width.
• Learning outcomes in this regime are strictly binary (instant generalization or outright failure), demonstrating that representability acts as a fundamental prior constraint on learning, distinct from optimization dynamics.
• Bottlenecked ReLU networks reveal a continuous path from "failure to memorize" (like the holomorphic network) to "memorization without generalization" and eventually to "grokking" as capacity increases.

Summary by alphaXiv.