Reproduce “Learning Curve Theory” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Marcus Hutter
Marcus Hutter, affiliated with DeepMind, introduces a minimalist theoretical model explaining empirically observed power-law learning curves in machine learning. The model demonstrates that unbounded feature spaces and Zipf-distributed data lead to n^(-β) performance scaling, including exponents less than 0.5, and shows why time-averaged learning curves appear robust despite high instantaneous variance.
Key insights
- The 'unbounded complexity' of the feature space is essential for the model to exhibit n^(-β) power-law learning curves across a broad range of β values, including those less than 0.5.
- A direct theoretical link shows that Zipf-distributed data (where feature probabilities `θ_i ∝ i^(-(α+1))`) yields power-law learning curves with exponent β = α / (1 + α).
- While instantaneous error observations are highly noisy, time-averaged learning curves are robust, with their signal-to-noise ratio decreasing to zero as the number of data points averaged increases.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.