Reproduce “Judge Reliability Harness: Stress Testing the Reliability of LLM Judges” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Sunishchal Dev, Andrew Sloan, Joshua Kavner, Nicholas Kong, Morgan Sandler

The Judge Reliability Harness (JRH), an open-source library, is introduced to systematically stress-test the reliability of large language model (LLM) judges in AI evaluation. This work reveals that LLM judge reliability varies significantly across tasks and perturbation types, demonstrating that smaller open models can achieve comparable reliability to larger, more costly frontier models for certain tasks.

Key insights

  • LLM judge reliability is highly task-dependent, with models showing greater robustness on binary classification tasks but fragility on multi-class ordinal scoring.
  • Judges are often less robust to superficial formatting changes (format invariance) than to semantically neutral paraphrases, indicating sensitivity to non-content variations.
  • Some smaller or moderately sized open models can achieve reliability comparable to or exceeding that of premium frontier judges at significantly lower inference costs.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.