Reproduce “Selective Differential Privacy for Language Modeling” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Weiyan Shi, Aiqi Cui, Evan Li, Ruoxi Jia, Zhou Yu

Selective-DPSGD applies selective differential privacy to policy-marked sensitive tokens during language-model training, using ordinary SGD on unmarked spans and clipped, noised gradients and hidden states on marked spans. On WikiText-2, it reports test perplexity of 160.04 ± 4.86, versus 305.86 ± 3.00 for DPSGD, at similar reported privacy budgets, though clipping thresholds differ. Its formal guarantee covers policy-marked attributes, not missed secrets.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.