Reproduce “Residual Alignment: Uncovering the Mechanisms of Residual Networks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jianing Li, Vardan Papyan
Researchers from the University of Toronto uncovered "Residual Alignment" (RA), a phenomenon characterizing the structured organization of internal representations in trained Residual Networks. This work identifies four key properties of RA, including equispaced intermediate representations and aligning low-rank Jacobians with singular values that scale inversely with depth, providing a geometric explanation for ResNets' effectiveness.
Key insights
- Well-trained ResNets exhibit "Residual Alignment" (RA), a set of four consistent properties in their internal representations.
- RA includes equispaced intermediate representations, singular vector alignment across layers, low-rank Residual Jacobians, and inverse scaling of singular values with depth.
- Skip connections are essential for the emergence of Residual Alignment, and RA co-occurs with Neural Collapse, suggesting a link to generalization.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.