Reproduce “An exactly solvable model for emergence and scaling laws in the multitask sparse parity problem” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yoonsoo Nam, Nayara Fonseca, Seok Hyeong Lee, Chris Mingard, Ard A. Louis
A theoretical study from the University of Oxford introduces an exactly solvable multilinear model to explain the phenomena of emergence and scaling laws in deep learning, demonstrating how these behaviors arise naturally from the learning of power-law distributed, orthogonal skills and accurately predicting skill acquisition in neural networks.
Key insights
- Emergence of skills can arise naturally from the multilinear (product of parameters) structure of the model, exhibiting sigmoidal growth in skill strength.
- The orthogonality of skills allows for decoupled learning dynamics, explaining how continuous overall loss decrease can result from discrete, sequential skill acquisitions, especially with power-law distributed skill frequencies.
- Feature learning in neural networks can be characterized by simple calibration parameters, suggesting a fundamental, quantifiable relationship between an NN's internal representations and skill acquisition.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.