Reproduce “End-to-End Differential Privacy in Training Deep Neural Network Classifiers” on OpenResearch

About this paper

Huaiyuan Rao, Calvin Hawkins, Alexander Benvenuti, Matthew Hale

A new framework for differentially private deep neural network training offers input-only privacy by perturbing softmax outputs with a Dirichlet mechanism. This method achieves substantially higher model accuracy, for instance, an 88.17% accuracy at ϵ=4 on CIFAR10, representing a 45.31% reduction in classification error, while also reducing training time compared to existing DP-SGD approaches.

Key insights

• The inherent compactness of the unit simplex (where softmax outputs reside) provides L2- and L-infinity sensitivity bounds, which are crucial for precisely calibrating the Dirichlet privacy mechanism.
• The Dirichlet mechanism primarily rescales the magnitude of the expected gradients during training, while preserving their direction, allowing for effective learning rate calibration and stable optimization.
• This work presents the first subsampled privacy analysis for the Dirichlet mechanism within the R´enyi differential privacy framework, enabling tighter privacy bounds for iterative training processes.

Summary by alphaXiv.