Reproduce “JEV as a Judge for Agent Trace Security: An Empirical Comparison with Generative LLM Judges” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Zhiqiang Wang, Yichao Gao

The paper evaluates JEV, a typed-decision judge that reviews completed tool-using agent traces using risk scores, authorization, intrinsic safety, and probabilities rather than free-form rationales. Across 5,219 trajectories, JEV’s mean positive-class F1 is 77.8%, versus 74.1% for GLM-5.2, with 0.99-second versus 33.29-second median successful-call latency; GLM leads on two datasets, and causality is unestablished.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.