Reproduce “S2A: A Unified Framework for Parameter and Memory Efficient Transfer Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tian Jin, Enjun Du, Changwei Wang, Wenhao Xu, Ding Luo
S2A (Structure to Activation) introduces a unified framework for parameter-efficient transfer learning that also drastically reduces training-time GPU memory footprint. It achieves memory reductions up to 9.7x compared to full fine-tuning while maintaining competitive accuracy across various backbone architectures and tasks.
Key insights
- By careful architectural design, gradient computations for specific parametric components (like bias vectors or low-rank matrices in LRP) can be made independent of the original layer input activations, enabling memory release.
- Activations of non-parametric layers can be effectively quantized to low bit-widths (e.g., 4-bit for Softmax/GELU, 1-bit for ReLU) with minimal accuracy loss, leveraging their specific derivative properties and value ranges.
- Combining activation-efficient parametric modules with targeted non-parametric activation quantization provides a more holistic and effective approach to memory-efficient PETL than either strategy alone.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.