Reproduce “Automatic Clipping: Differentially Private Deep Learning Made Easier and Stronger” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhiqi Bu, Yu-Xiang Wang, Sheng Zha, George Karypis
This research introduces automatic clipping, a mechanism that eliminates the need to tune the gradient clipping threshold (R) in differentially private deep learning, achieving comparable or superior model accuracy and stability across diverse vision and language tasks.
Key insights
- For non-adaptive optimizers like SGD, tuning R is mathematically equivalent to scaling the learning rate, implying that R's effect is absorbed by the learning rate hyperparameter.
- In adaptive optimizers (e.g., Adam), the R factor cancels out in the update rule, rendering these optimizers largely insensitive to the choice of R.
- Introducing a positive stability constant `γ` (AUTO-S) is essential for convergence to zero gradient norms in non-convex optimization, as it preserves magnitude information for smaller gradients and prevents a 'lazy region' where updates stall.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.