Reproduce “Compress to Impress: Efficient LLM Adaptation Using a Single Gradient Step on 100 Samples” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shiva Sreeram, Alaa Maalouf, Pratyusha Sharma, Daniela Rus
Researchers from MIT CSAIL developed an efficient, training-free method to adapt large language models (LLMs) by intelligently pruning specific weight matrix components. The approach achieved over 50x speedup on GPT-J and 22.2x on RoBERTa, and up to a 24 percentage point accuracy improvement on specific tasks compared to prior training-free techniques, utilizing as few as 100 calibration samples.
Key insights
- Singular value gradients derived from a single backward pass can effectively identify weight matrices that benefit most from rank reduction.
- Only about 100 labeled examples are sufficient for both guiding the matrix selection and evaluating the performance of compressed models.
- Weight matrices are better represented as mixtures of low-dimensional subspaces, making multi-subspace (clustered) SVD more effective for denoising and accuracy gains than a single global SVD.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.