Reproduce “Understanding Scaling Laws with Statistical and Approximation Theory for Transformer Neural Networks on Intrinsically Low-dimensional Data” on OpenResearch

About this paper

Alex Havrilla, Wenjing Liao

Researchers at Georgia Institute of Technology developed statistical and approximation theories for transformer neural networks, demonstrating how generalization error scales with data size and network size when data resides on a low-dimensional manifold. Their framework explicitly quantifies these power-law exponents based on the data's intrinsic dimension and function regularity, with data scaling predictions empirically aligning well with observations from large language models.

Key insights

• Scaling law exponents for transformer generalization error are explicitly linked to the intrinsic dimension of the data ($d$) and the Hölder continuity of the target function ($\beta$).
• Transformers can approximate $\beta$-Hölder continuous functions on $d$-dimensional manifolds with an $O(\log(d))$ network depth, independent of approximation accuracy.
• The work introduces novel mathematical tools, including a new covering number calculation for transformer networks and an "Interaction Lemma" for sparse token interactions.

Summary by alphaXiv.