Reproduce “Learning in High Dimension Always Amounts to Extrapolation” on OpenResearch

About this paper

Randall Balestriero, Jerome Pesenti, Yann LeCun

A study by Facebook AI Research and NYU demonstrates that current machine learning models consistently operate in an extrapolation regime when processing high-dimensional data, challenging the long-held belief that good generalization requires interpolation within the training data's convex hull. This work reveals that for dimensions above 100, the probability of new samples falling within the training convex hull approaches zero, yet models still perform well.

Key insights

• Interpolation, defined as falling within the convex hull of training data, almost never occurs in high-dimensional spaces (d > 100) for realistically sized datasets.
• The critical factor determining the probability of interpolation is the dimension of the smallest affine subspace containing the data (d*), not the intrinsic manifold dimension or ambient dimension.
• Extrapolation, under the standard geometric definition, is not an indicator of poor generalization performance, as high-performing deep learning models consistently operate in this regime.

Summary by alphaXiv.