Reproduce “Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Wuyang Chen, Xinyu Gong, Zhangyang Wang

The TE-NAS framework offers a training-free approach to Neural Architecture Search, utilizing the Neural Tangent Kernel and linear regions to assess network trainability and expressivity at initialization. This methodology allows for the discovery of competitive architectures on ImageNet in only four GPU hours, drastically reducing search costs compared to existing methods.

Key insights

  • The condition number of the Neural Tangent Kernel (NTK) negatively correlates with test accuracy (lower indicates better trainability), while the number of linear regions positively correlates with test accuracy (higher indicates greater expressivity).
  • Combining the ranked values of both trainability (NTK) and expressivity (linear regions) indicators yields a stronger predictive correlation with an architecture's final test accuracy.
  • The search process initially prioritizes improving network trainability, then fine-tunes architectures to retain expressivity, indicating a balanced approach to architectural quality.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.