Reproduce “Robustness Emerges Early in Training Dynamics, but Is Not Preserved” on OpenResearch

About this paper

Jiangang Yang, Wenhui Shi, Lu Hu, Jing Xing, Jian Liu

This paper introduces "robustness fading," a phenomenon where robust representations emerge early in shallow network layers but degrade during standard optimization. It proposes two parameter-free strategies, Early-Phase Stabilization (EPS) and Asymmetric Weight Reversion (AWR), which effectively preserve these early-emergent robust priors, enhancing model robustness against natural corruptions across various vision tasks and architectures.

Key insights

• Shallow layers of deep neural networks spontaneously develop robust representations and a flatter loss landscape early in their training dynamics.
• These valuable early-emergent robust properties in shallow layers are not preserved and undergo a process of "robustness fading" as standard training continues towards convergence.
• Strategic, parameter-free interventions in the training trajectory of shallow subnetworks can effectively counteract this fading phenomenon, maintaining robust priors and improving overall model resilience.

Summary by alphaXiv.