Reproduce “GRASP: GRouped Activation Shared Parameterization for Parameter-Efficient Fine-Tuning and Robust Inference of Transformers” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Malyaban Bal, Abhronil Sengupta
GRASP adapts pretrained transformers with fewer parameters by sharing scale and shift values across groups of hidden dimensions while freezing the weights. StochGRASP learns distributions over weight perturbations and trains against simulated noise. On SST-2 with RoBERTa-large, StochGRASP retains at least 86% accuracy versus 61.5% for deterministic GRASP under Gaussian noise, but the stated noise level conflicts with Figure 6.
Key insights
- Grouped modulation of activation functions can achieve high parameter efficiency (O(n x K) parameters) for fine-tuning large transformers without introducing additional matrix multiplications during training.
- Reducing the group size in GRASP reveals structured, multimodal distributions in learned parameters, suggesting that PEFT can be interpreted as learning underlying perturbation distributions.
- By modeling fine-tuning perturbations stochastically (e.g., as Gaussian distributions), models can be inherently made robust against hardware-level noise and variability during inference.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.