Reproduce “Learning curves theory for hierarchically compositional data with power-law distributed features” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Francesco Cagnetta, Hyunmo Kang, Matthieu Wyart

This research investigates how hierarchical data structure and power-law feature distributions jointly influence neural scaling laws in deep learning, using an extended Random Hierarchy Model (RHM). It demonstrates that for classification tasks, power-law distributed features directly dictate the asymptotic learning curve exponent, while for next-token prediction, the hierarchical structure primarily governs the asymptotic scaling, with feature distribution controlling the residual loss.

Key insights

  • The origins of neural scaling laws are task-dependent: for classification, the asymptotic power-law exponent is determined by the Zipfian feature distribution, while for next-token prediction, it is predominantly driven by the hierarchical data structure.
  • Even when classification learning curves yield exponents identical to simpler memorization theories, the hierarchical RHM framework indicates genuine generalization through rule inference, rather than rote memorization.
  • For next-token prediction, the hierarchical structure sets the asymptotic learning curve exponent, while the Zipfian feature distribution primarily influences the local details of learning and the ultimate residual error.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.