Reproduce “JudgeMoE: Distributional Aggregation for LLM-as-a-Judge” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yiqi Liu, Joseph James, Yang Wang, Kun Zhao, Chenghao Xiao, Chenghua Lin
JudgeMoE combines cached score distributions from multiple language model judges using example-specific weights and a tempered log-opinion pool, then predicts an expected score. Across 16 dataset–scale conditions, it improved mean Spearman correlation over the strongest local single judge by 0.0393, winning in 12/16 conditions (one-sided p = 0.0091). On the original 10-cell benchmark, stronger fixed-pooling comparisons remained inconclusive.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.