Reproduce “Generalization Properties of NAS under Activation and Skip Connection Search” on OpenResearch

About this paper

Zhenyu Zhu, Fanghui Liu, Grigorios G Chrysos, Volkan Cevher

Researchers at EPFL established a generalized Neural Tangent Kernel (NTK) framework to provide theoretical guarantees for the generalization properties of multi-layer Neural Architecture Search (NAS) encompassing both diverse activation functions and skip connections. This framework informed Eigen-NAS, a train-free NAS algorithm demonstrating competitive performance on benchmark datasets.

Key insights

• The generalization performance of a NAS-discovered architecture is inversely correlated with the minimum eigenvalue of its Neural Tangent Kernel (NTK); architectures with larger minimum eigenvalues are predicted to generalize better.
• The choice of activation function, particularly in the first layer, and the presence of skip connections significantly influence the NTK minimum eigenvalue and, consequently, the network's generalization ability.
• Theoretical analysis revealed that ReLU's minimum eigenvalue bound remains constant with depth, LeakyReLU's increases, while Sigmoid, Tanh, and Swish's decrease, providing theoretical justification for common activation function preferences in deep networks.

Summary by alphaXiv.