Reproduce “Just Ask Jev: Reinforcement Learning for Calibrated Decisions as a Zero-Shot Detector of AI Alignment Failures” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ruoqi Guo, Yi Liu, Gelei Deng, Yuekang Li, Lida Zhao, Yutao Wu, Simin Chen, Ying Zhang and 1 more
The paper evaluates Jev, a model that answers multiple typed questions about one interaction state in a single call, as a detector of benchmark-labeled language-model failures. RLCDAlignBench covers 44 benchmarks, ten failure types, and 7,193 instances. A generic probability question reaches median AUROC 0.886 on 31 benchmarks, but probabilities are often miscalibrated within individual benchmarks.
Key insights
- A simple generic yes/no probability question is already a strong zero-shot readout; more elaborate targeted wording offers only a small, statistically uncertain median AUROC gain.
- Context helps selectively: label-defining information can substantially boost benchmark scores, but this may reflect access to the benchmark’s answer key rather than broader failure-detection ability.
- Jev’s scores are better suited to ranking than to universally calibrated probabilities, and disagreements with detectors can expose problems in benchmark definitions or labels.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.