Reproduce “NATS-Bench: Benchmarking NAS Algorithms for Architecture Topology and Size” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Xuanyi Dong, Lu Liu, Katarzyna Musial, Bogdan Gabrys

Researchers from the University of Technology Sydney introduce NATS-Bench, a comprehensive benchmark that pre-computes the performance of 15,625 topological and 32,768 size-varying neural architectures across three datasets. It addresses the lack of comparability and reproducibility in Neural Architecture Search by providing consistent evaluation data, significantly accelerating algorithm assessment.

Key insights

  • Architecture rankings for topology generally remain consistent across datasets, while optimal architecture size is often dataset-specific, highlighting different generalization behaviors.
  • A strong positive correlation exists between validation and test accuracy within the benchmark, validating its utility for guiding NAS algorithm development.
  • Transferring top-performing architectures directly from one dataset to another is often ineffective, calling for more sophisticated, transferable NAS algorithms.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.