Reproduce “From Judgment Quality to Downstream Utility: Rethinking LLM-as-a-Judge for Open-Ended Tasks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zheng Zhang, Lufei Li, Xinyue Tan, Yuanhao Zeng, Ziwei Shan, Yexin Li, Kan Ren
The authors evaluate rubric-based language-model judges for training and inference-time guidance on open-ended medical and scientific questions. They vary rating format, critique order, and batching, then test selection, revision, and beam search with frozen policies. Rating improves training scores in 10 of 16 comparisons, while Best-of-N selection yields mean gains of 7.8 points at N=4 and 17.2 at N=40, with added computation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.