Reproduce “The Multiscale Single-Index Model: A Stylized Model for Hierarchical Feature Learning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Joan Bruna

The authors analyze simultaneous online spherical SGD for a multiscale tensor model with Gaussian data. An Edgeworth expansion resolves its nonlinear structure, and the updates optimize teacher–student correlation. Under fixed-depth assumptions and favorable initialization, the theorem gives near-perfect recovery with O(d^(K−1)) samples. At K=4, d=64, 3/10 correlation runs versus 0/10 MSE runs met the overlap criterion, a finite-run result.

Key insights

  • The MSIM's Wiener chaos exhibits a precise, finite-rank perturbation structure, decomposing into a hierarchy of terms with decreasing magnitudes that are analytically accessible through Edgeworth expansions.
  • Approximation lower bounds demonstrate that shallow neural networks cannot efficiently learn MSIM targets, providing a formal quantification of the necessity of depth for this class of hierarchical functions.
  • Standard backpropagation via online SGD on the correlation objective efficiently recovers planted features across all layers simultaneously, validating its practical success for deep hierarchical models.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.