Reproduce “Can Programs Replace LLM Judges? Executable Program Judges for Scalable and Efficient Relevance Labeling” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Eugene Agichtein, Charles L. A. Clarke, Kaustubh Dhole, William Stern
ExecJudges evaluates retrieval results by having an LLM write topic-specific programs that apply grading rules across a document pool, without further LLM grading calls. Three programs check evidence in each passage, and their grades are averaged and rounded down. On TREC DL 2021, agreement with NIST labels was 0.653, versus 0.635 for listwise Claude Sonnet 5.5; results varied across collections.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.