Reproduce “Differentially Private Bias-Term Fine-tuning of Foundation Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhiqi Bu, Yu-Xiang Wang, Sheng Zha, George Karypis
Researchers at Amazon AI, along with the University of California, San Diego, developed Differentially Private Bias-Term Fine-tuning (DP-BiTFiT), a method for applying differential privacy to large pre-trained models by exclusively training bias parameters. This approach achieved up to 30 times faster training and 8 times less memory usage than previous DP fine-tuning techniques, while maintaining comparable or superior accuracy on various NLP and CV tasks.
Key insights
- Bias gradients can be computed in an 'activation-free' manner during back-propagation, offering a fundamental pathway to eliminate the largest source of memory and computational overhead in differentially private training.
- Focusing DP-SGD solely on bias terms allows for extremely high parameter efficiency (around 0.1% of parameters) without significant loss in model utility compared to training more parameters.
- The computational overhead of DP-BiTFiT is largely independent of the input feature dimension, making it exceptionally efficient for high-dimensional data, unlike prior DP methods whose costs scale with feature dimension.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.