Reproduce “Equivariant Differentially Private Deep Learning: Why DP-SGD Needs Sparser Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Florian A. Hölzl, Daniel Rueckert, Georgios Kaissis
Researchers from the Technical University of Munich integrated Equivariant Convolutional Networks (ECNNs) with Differentially Private Stochastic Gradient Descent (DP-SGD) to address privacy-utility and accuracy-efficiency trade-offs. This method achieved up to 9.2% higher accuracy on imaging benchmarks while using 35 times fewer parameters and reducing computation time by 85%, alongside improved model calibration.
Key insights
- Networks exhibiting "designed sparsity," such as ECNNs, can overcome the privacy-utility trade-off more effectively and efficiently than brute-force computational scaling with over-parameterized models.
- ECNNs lead to more efficient parameter utilization and sparser, more effective gradient vectors during DP-SGD training, implying that a greater proportion of parameters actively contribute to learning and information transfer.
- Discrete symmetry groups, particularly D₄, offer the best balance of expressiveness and efficiency for ECNNs in DP-SGD, significantly reducing the need for computationally expensive data augmentation multiplicity.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.