Reproduce “Eigenspectrum Analysis of Neural Networks without Aspect Ratio Bias” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yuanzhe Hu, Kinshuk Goel, Vlad Killiakov, Yaoqing Yang
This research introduces the Fixed-Aspect-Ratio Matrix Subsampling (FARMS) method, designed to mitigate the aspect ratio bias in eigenspectrum analysis of neural network weight matrices. FARMS enables more accurate layer-wise quality assessment by normalizing Empirical Spectral Densities, which led to a reduction in LLaMA-7B perplexity from 96.02 to 79.42 for 0.8 sparsity in pruning tasks and improved image classification accuracy for VGG19 models to 74.34%.
Key insights
- The aspect ratio of neural network weight matrices is a critical confounding factor in the interpretation of their Empirical Spectral Densities, previously overlooked in many HT-SR applications.
- Normalizing for this architectural bias through fixed-aspect-ratio subsampling yields a more accurate and robust metric of layer-wise training quality, correcting previous misdiagnoses.
- Improved accuracy in quantifying layer heavytailness directly translates to measurable performance gains in various downstream tasks, including LLM pruning, image classification training, and scientific machine learning fine-tuning.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.