Reproduce “Towards Understanding Generalization in DP-GD: A Case Study in Training Two-Layer CNNs” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Zhongjie Shi, Puyu Wang, Chenyang Zhang, Yuan Cao

Researchers from Georgia Institute of Technology and The University of Hong Kong identified a binary classification task where Differentially Private Gradient Descent (DP-GD) surpasses standard Gradient Descent (GD) in generalization performance for two-layer CNNs. Their analysis shows that DP-GD's injected noise regularizes learning, preventing noise memorization and achieving up to 20% higher test accuracy than GD in high noise environments, while also providing privacy guarantees.

Key insights

  • Standard GD, under specific low SNR conditions, primarily memorizes noise components in data, leading to a low training loss but poor generalization performance.
  • The deliberately injected Gaussian noise in DP-GD serves as a regularization mechanism, enabling signal learning components to surpass activation thresholds and effectively dominate the learning process.
  • DP-GD can achieve both strong privacy guarantees and superior generalization performance compared to standard GD in certain learning scenarios, reframing the privacy-utility relationship.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.