Reproduce “Harnessing large-language models to generate private synthetic text” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Alexey Kurakin, Natalia Ponomareva, Umar Syed, Liam MacDermed, Andreas Terzis

Researchers at Google developed a methodology leveraging differentially private fine-tuning of large language models with parameter-efficient techniques like LoRA to generate high-utility synthetic text data. This approach yields downstream classification performance competitive with or superior to directly trained differentially private models, while ensuring robust privacy guarantees through pre-training data de-duplication.

Key insights

  • Parameter-efficient fine-tuning, particularly LoRA, offers a superior privacy-utility trade-off compared to full fine-tuning for differentially private text generation.
  • The standard Prefix-LM objective is effective for DP fine-tuning, eliminating the need for custom or augmented loss functions found in previous work.
  • De-duplicating the LLM's public pre-training data against sensitive downstream datasets is critical to prevent privacy leakage and ensure the integrity of DP guarantees.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.