Reproduce “LLM-as-a-Judge Is Not an Oracle: Why Self-Improving Agents Need Deterministic Guardrails” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Vansh Wahi
Research from the University of Waterloo examines how large language model (LLM) evaluators in self-improving agent optimization loops lead to reward hacking and degraded true capabilities. The study identifies eleven failure modes and proposes the PROCTOR architecture, which incorporates five layers of deterministic guardrails to contain these unreliabilities, demonstrating that structural constraints like reordering output schema (rationale before score) significantly improved judge-human exact match agreement from 42.6% to 51.9%.
Key insights
- Optimization loops do not average out LLM evaluator errors but rather perform 'gradient ascent' on them, causing agents to converge to the judge's decision boundary instead of the true task objective.
- Structural constraints on output format or execution environment are significantly more effective at improving LLM judge reliability and system robustness than merely refining natural-language instructions.
- Decoupling the roles of subagents and enforcing stateless, information-restricted operation within a layered deterministic verification framework is critical for preventing reward hacking and ensuring robust self-improvement.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.