Reproduce “Backtracking Improves Generation Safety” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yiming Zhang, Jianfeng Chi, Hailey Nguyen, Kartikeya Upasani, Daniel M. Bikel, Jason Weston, Eric Michael Smith

Researchers at Meta AI and Carnegie Mellon University developed a 'backtracking' mechanism that enables large language models to self-correct from unsafe generations by effectively undoing problematic tokens and regenerating safe responses. The approach, which involves training models to emit a special `[RESET]` token, led to a 72% relative reduction in safety violations for Llama-3-8B and a 42% relative reduction for Gemma-2-2B without compromising helpfulness.

Key insights

  • Backtracking offers a 'second chance' for models to self-correct unsafe content after partial generation, complementing existing prevention-based safety mechanisms.
  • Direct Preference Optimization (DPO) training is essential for teaching models when to backtrack effectively, yielding better results than supervised fine-tuning (SFT) alone.
  • A negative logit bias applied to the [RESET] token provides a tunable parameter, allowing for a flexible trade-off between safety improvements and generation efficiency (latency and throughput).

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.