Reproduce “Same Attack, Different Judge: A First Controlled Adversarial Comparison of Jev, LLM Judges, and Purpose-Built Safety Classifiers” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xiang Li, Aichun Huang, Mingming Zhang, Zheli Liu
The paper compares Jev with LLM judges as safety gates for attacker-controlled content, giving systems the same content and routing policy in their expected formats. Jev’s attack success rate was 9% on a harmful-request test, versus 56% for GPT-6 Astra, but forged moderation fields bypassed Jev on phishing. Phishing evidence covers one email corpus and single-turn calls.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.