Reproduce “Instance-Optimal Estimation with Multiple LLM Judges on a Budget” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Junghyun Lee, Sanghwa Kim, Yassir Jedra, Alexandre Proutière, Se-Young Yun
The authors allocate a fixed query budget across judges and prompt–response pairs by estimating pair–judge score variances and favoring better cost–reliability trade-offs. Adaptive allocation lowers L2 error than uniform allocation in shown experiments. Theoretically, it matches the oracle’s leading error rate for bounded, unbiased scores with known judge costs and a shared true score per pair.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.