Reproduce “Large Language Models Can Be Strong Differentially Private Learners” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Xuechen Li, Florian Tramèr, Percy Liang, Tatsunori Hashimoto

Research from Stanford and Google demonstrates that large pre-trained language models can be successfully fine-tuned with strong differential privacy guarantees (e.g., epsilon 3 or 8) across various NLP tasks. The work identifies optimal hyperparameter settings and introduces "ghost clipping" for memory-efficient private training, achieving performance comparable to or sometimes exceeding non-private baselines.

Key insights

  • Differential privacy is not inherently incompatible with high-performing large language models; careful optimization of existing techniques is key.
  • Larger pre-trained models are more robust to the noise introduced by differential privacy, often leading to better private fine-tuning performance, a direct contradiction to prior assumptions.
  • Optimal hyperparameters for DP training often differ from non-private training, with large batch sizes, large learning rates, and small clipping norms proving effective for DP-Adam.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.