Reproduce “GradSafe: Detecting Jailbreak Prompts for LLMs via Safety-Critical Gradient Analysis” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yueqi Xie, Minghong Fang, Renjie Pi, Neil Gong
Researchers at HKUST, the University of Louisville, and Duke University developed GradSafe, an efficient gradient-based method for detecting jailbreak prompts in Large Language Models. GradSafe-Zero achieved an AUPRC of 0.755 on ToxicChat, surpassing Llama Guard's 0.635 in zero-shot detection, and GradSafe-Adapt matched Llama Guard's full-data performance using only 20% of the training data.
Key insights
- Jailbreak prompts, when paired with a compliance response, induce distinct and consistent gradient patterns on specific 'safety-critical parameters' within the LLM.
- Focusing on these critical parameters, rather than the entire model, allows for efficient and effective detection without requiring extensive finetuning or large labeled datasets.
- The effectiveness of gradient-based detection is tied to the underlying safety alignment of the base LLM, demonstrating that the method leverages the model's inherent safety mechanisms.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.