Reproduce “Measuring memorization in RLHF for code completion” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Aneesh Pappu, Billy Porter, Ilia Shumailov, Jamie Hayes

Researchers at Google DeepMind and Google empirically analyzed memorization risks in Large Language Models aligned via Reinforcement Learning with Human Feedback (RLHF) and direct preference learning for code completion. The study found that RLHF significantly mitigates memorization of reward model training data, showing 0% memorization for PII-like paths, compared to direct preference learning methods like IPO which exhibited substantially higher memorization rates up to 19.5%.

Key insights

  • The multi-stage nature of RLHF, particularly with its KL regularization, acts as a protective mechanism against memorization of sensitive preference data, especially when used in reward model training.
  • Direct preference learning algorithms like IPO, while simplifying the alignment process, introduce significantly higher risks of memorization of preference data compared to RLHF.
  • The KL regularization coefficient (a) in RLHF plays a crucial role in preventing the RL fine-tuned model from over-optimizing to and memorizing reward model training data, while also modulating the persistence of initial fine-tuning memorization.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.