Reproduce “Does the Unsafe Gradient Survive a Conversation? On the Fragility of Gradient-Based Jailbreak Detection in Multi-Turn Dialogue” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Omar Sheta, Rinku Deuja, Hadi Masoudi, Minghong Fang
The paper adapts GradSafe to detect unsafe intent distributed across user turns by scoring windows with a gradient-based signal and using the highest score. On Llama 3.1 8B Instruct, single-turn windows outperform three-turn windows against WildChat benign conversations, with ROC-AUC 0.8101 versus 0.7614. However, synthetic calibration transfers poorly, Crescendo attacks rank near or below chance, and results depend on model.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.