Reproduce “DP-aware AdaLN-Zero: Taming Conditioning-Induced Heavy-Tailed Gradients in Differentially Private Diffusion” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tao Huang, Jiayang Meng, Xu Yang, Chen Hou, Hong Chen
A novel conditioning mechanism, DP-aware AdaLN-Zero, is introduced for differentially private diffusion models. It effectively mitigates heavy-tailed gradients induced by diverse conditioning signals through deterministic, per-block constraints in the forward pass, leading to improved utility in tasks such as time-series imputation and forecasting.
Key insights
- Heterogeneous conditional contexts can be a direct source of heavy-tailed gradients, significantly impacting the effectiveness of DP-SGD in conditional generative models.
- Architectural interventions in the model's forward pass, specifically by constraining the conditioning pathway, can effectively tame gradient spikes and improve the privacy-utility trade-off in DP-trained models.
- Smooth bounding operators for AdaLN parameters perform better than hard truncation, indicating the importance of maintaining smooth gradient flow while controlling magnitude.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.