Reproduce “Shape it Up! Restoring LLM Safety during Finetuning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
ShengYun Peng, Pin-Yu Chen, Jianfeng Chi, Seongmin Lee, Duen Horng Chau
Researchers from Georgia Tech, IBM Research, and Meta introduced Dynamic Safety Shaping (DSS), a framework employing token-level safety signals (STAR scores) to restore Large Language Model safety during finetuning. This approach consistently outperforms static baselines, achieving up to 89.42% safety on AdvBench in worst-case scenarios, while preserving model capabilities and showing robustness against various adversarial attacks.
Key insights
- Moving from static, example-level safety decisions to dynamic, token-level assessments offers a more robust defense against safety degradation during finetuning.
- Existing safety guardrail models can be effectively repurposed to generate fine-grained, continuous safety signals, enabling adaptable and practical safety interventions.
- A theoretical guarantee demonstrates that `⋆DSS` can provably bound the harmfulness introduced during finetuning, controlling it relative to the original safety-aligned model.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.