Reproduce “On-Policy Distillation for LLM Safety: A Routing Approach to Template-Robust Realignment” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yongjian Guo, Wanlun Ma, Lingyu Shen, Xi Xiao, Sheng Wen
Routing-based On-Policy Distillation (ROPD) offers a framework for realigning compromised large language models by distilling output probabilities from dual teachers. This approach robustly restores safety without compromising specialized skills and shows increased resilience to prompt template mismatches compared to existing methods.
Key insights
- The enduring signature of an attack resides in the model’s output probability distribution, which exhibits a sharp disagreement between aligned and misaligned models across various templates, offering a robust target for repair.
- Decoupling safety and skill objectives through a dual-teacher routing mechanism enables effective safety realignment without significant degradation of the model's specialized abilities.
- Weight-level repairs, including ROPD, face an intrinsic robustness boundary, as re-eliciting harmful behavior is still possible through prompt rewriting, highlighting the conditional nature of current safety fixes.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.