Reproduce “Skip Connections Eliminate Singularities” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
A. Emin Orhan, Xaq Pitkow
The authors examine skip connections in deep fully connected ReLU networks. They propose skip inputs can keep hidden units active and distinguishable, reducing singularities, where parameter settings have indistinguishable effects on outputs. On CIFAR-100, hyper-residual networks train fastest and plain networks train slowest, alongside the reverse ordering in estimated Hessian degeneracy. This supports singularity reduction as a partial, not exclusive explanation.
Key insights
- Degeneracy in the loss landscape, quantifiable by the fraction of near-zero eigenvalues of the Hessian, inversely correlates with the speed and final accuracy of network training.
- Skip connections provide benefits that extend beyond merely favorable initialization or solely mitigating vanishing gradients, actively restructuring the loss landscape by moving parameters away from singular regions.
- The effectiveness of skip connections can be further enhanced by incorporating non-identity, orthogonal connections, which maximize unit disambiguation and reduce the probability of zero hidden unit responses.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.