Reproduce “A CaLLM Judge is a Better Judge: Mitigating Bias in LLM-as-a-judge” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Cesare Barbera, Luca Stradiotti, Lorenzo Cascioli, Lorenzo Perini, Niek Tax, Daniel Haimovich, Jesse Davis, Andrea Passerini and 1 more

CaLLM calibrates confidence in pairwise language-model judgments using representations of answer differences and human-preference calibration data. It applies multicalibration to adjust errors across groups, including constructed unspecified groups. Across five datasets, it had the highest accuracy on four and tied the best baseline on one, with lowest log loss on all five; these distributional results do not establish causal fairness.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.