Reproduce “Evaluating Efficient Performance Estimators of Neural Architectures” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xuefei Ning, Changcheng Tang, Wenshuo Li, Zixuan Zhou, Shuang Liang, Huazhong Yang, Yu Wang
A systematic evaluation assesses One-shot and Zero-shot Estimators for Neural Architecture Search across five diverse benchmarks, analyzing their performance, biases, and variances. The study found One-shot Estimators generally outperform Zero-shot counterparts and demonstrated that strategies like de-isomorphism sampling improved One-shot Estimator top-K precision by over 25 percentage points on some benchmarks.
Key insights
- One-shot Estimators (OSEs) generally yield better ranking quality than Zero-shot Estimators (ZSEs), with many ZSEs performing worse than basic structural metrics like parameter count.
- OSEs demonstrate biases, such as underestimating larger architectures and misestimating certain operation types, but their ranking quality improves with sufficient training.
- Parameter-level ZSEs often generate improper biases (e.g., favoring linear architectures due to gradient explosion sensitivity) and are largely insensitive to input data distribution.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.