Reproduce “SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Maithra Raghu, Justin Gilmer, Jason Yosinski, Jascha Sohl-Dickstein
Singular Vector Canonical Correlation Analysis (SVCCA) is introduced as a method to quantitatively compare representations within deep neural networks. This approach reveals that deep networks are often highly over-parameterized and demonstrates a "bottom-up" learning dynamic, which informed a "freeze training" strategy that improved generalization.
Key insights
- Deep neural networks are highly over-parameterized, with the effective intrinsic dimensionality of learned representations often significantly lower than the number of neurons.
- Network training exhibits a "bottom-up" dynamic, where lower layers stabilize their representations much earlier than higher layers during the learning process.
- SVCCA can quantitatively reveal how class-specific and semantic information forms and propagates throughout a network's architecture.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.