Reproduce “On the Power and Limitations of Random Features for Understanding Neural Networks” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Gilad Yehudai, Ohad Shamir

This paper from the Weizmann Institute of Science systematically explores the explanatory power of random features for understanding over-parameterized neural networks, formalizing how many recent theories align with this framework. It demonstrates that while random features can explain the learnability of low-degree polynomials, they fail to efficiently learn even a single ReLU neuron, indicating a fundamental gap in current theoretical explanations for deep learning's capabilities.

Key insights

  • Many recent theoretical explanations for over-parameterized neural networks, including those based on the Neural Tangent Kernel, implicitly rely on the 'random features' paradigm where early layers are effectively fixed.
  • While random features can explain the learnability of certain functions like low-degree polynomials, they fundamentally fail to explain the efficient learning of even a single ReLU neuron.
  • The discrepancy between the empirical learnability of simple ReLU neurons and the exponential cost predicted by random feature models indicates a significant gap in current deep learning theory regarding representation learning.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.