Reproduce “Gradient Cuff: Detecting Jailbreak Attacks on Large Language Models by Exploring Refusal Loss Landscapes” on OpenResearch

About this paper

Xiaomeng Hu, Pin-Yu Chen, Tsung-Yi Ho

Gradient Cuff introduces a training-free method to detect jailbreak attacks on large language models (LLMs) by exploring their "refusal loss landscape." The system identifies malicious queries by their distinct refusal loss functional values and larger gradient norms, reducing the average attack success rate on models like LLaMA-2-7B-Chat and Vicuna-7B-V1.5 from 76.7% to 25.7% while maintaining acceptable false positive rates.

Key insights

• Malicious jailbreak queries generate distinct characteristics in an LLM's refusal loss landscape, exhibiting both lower refusal probabilities and significantly larger gradient norms.
• Zeroth-order gradient estimation can effectively approximate the gradient of the non-differentiable refusal loss function, enabling the use of gradient information for detection without requiring full white-box gradient access.
• A combined detection approach, leveraging both the functional value of the refusal loss and its gradient norm, provides a balanced and robust defense against diverse jailbreak attacks while preserving LLM utility.

Summary by alphaXiv.