Reproduce “Judge, Retrieve, or Abstain: Uncertainty-Guarded LLM Judging with Provable Risk Guarantees” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Sher Badshah, Ali Emami, Hassan Sajjad
A framework for LLM judges provides provable risk guarantees for objective, factual evaluations, controlling the False Discovery Rate among accepted verdicts. It employs an adaptive two-mode routing system that strategically uses web retrieval to significantly increase evaluation coverage.
Key insights
- The framework provides a formal, finite-sample guarantee on the False Discovery Rate of accepted LLM verdicts, offering statistically rigorous control over evaluation error rates.
- Adaptive retrieval significantly enhances evaluation coverage by routing initially uncertain instances to a tool-augmented mode, effectively converting potential abstentions into reliable verdicts and optimizing resource use.
- The Bernoulli structure required for Clopper-Pearson bounds extends to multi-mode sequential routing, providing a robust theoretical foundation for integrating risk-controlled selective prediction with adaptive decision-making.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.