Reproduce “DP-BART for Privatized Text Rewriting under Local Differential Privacy” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Timour Igamberdiev, Ivan Habernal
DP-BART, a Local Differential Privacy (LDP) text rewriting system based on a pre-trained BART model, enhances the privacy/utility trade-off by incorporating clipping-by-value and an iterative neuron pruning mechanism. The system consistently achieves higher utility for downstream tasks at stronger privacy levels compared to previous LDP text rewriting methods, notably reducing the required noise scale for a given privacy budget.
Key insights
- Clipping by value, applied to individual dimensions of a transformer's encoder output, effectively enables LDP for high-dimensional representations without destroying pre-trained knowledge, offering a viable alternative to clipping by norm.
- Systematic neuron pruning in the encoder based on decoder attention weights is an effective strategy to reduce the dimensionality of the privatized vector, directly translating to a substantial decrease in the amount of noise needed for a given privacy budget.
- Training a model with DP noise, after pruning, increases its robustness to privacy-preserving perturbations, allowing it to better reconstruct meaningful information from noisy representations and thereby enhance utility.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.