Reproduce “Random matrix analysis of deep neural network weight matrices” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Matthias Thamm, Max Staats, Bernd Rosenow
Researchers at the Institut für Theoretische Physik, Universität Leipzig, utilized Random Matrix Theory (RMT) to analyze the weight matrices of fully trained deep neural networks, demonstrating that while the bulk of the weights exhibit random characteristics, learned information is concentrated in the largest singular values and their corresponding eigenvectors. Their comprehensive analysis also showed that RMT effectively differentiates between various learning regimes and rigorously concluded that singular value distributions do not generically follow power-law tails, refining previous understandings.
Key insights
- A substantial portion of the weight matrices in fully trained deep neural networks retains statistical properties consistent with random matrices, particularly in the bulk of the singular value spectra.
- Learned information is primarily encoded in the largest singular values and their associated eigenvectors, which show significant deviations from RMT predictions, indicating structure and localization.
- Random Matrix Theory can effectively differentiate between 'lazy' and 'rich' learning regimes, offering a potential diagnostic tool for understanding and monitoring training dynamics.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.