Reproduce “SAFEGuard: Detect Optimization-Based Jailbreak Attacks Through Harmful Semantic Analysis and Fluency Measurement” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Quoc Viet Vo, Trung Le, Damith C. Ranasinghe, Ehsan Abbasnejad
SAFEGuard introduces a unified framework to detect optimization-based jailbreak attacks against Large Language Models by integrating a hybrid fluency measurement with gradient matching for semantic analysis. This framework achieved an average jailbreak detection rate of 99.8% on Llama-2-7B-Chat, significantly outperforming existing baselines while maintaining moderate benign refusal rates.
Key insights
- The combination of JSD with perplexity offers a more stable and robust fluency metric, effectively addressing perplexity anomalies in benign prompts while maintaining sensitivity to adversarial token injections.
- Neither fluency-based detection nor gradient matching alone is sufficient, as they exhibit complementary strengths against different types of optimization-based jailbreak attacks.
- High-fluency malicious prompts tend to maintain embeddings close to harmful samples (generating strong gradient signals), while low-fluency, obfuscated prompts can drift from these regions (weakening gradient signals), necessitating both detection modalities.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.