Reproduce “The Judge Can Debias Itself: Awareness and Self-Debiasing for Reliable LLM-as-a-Judge” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Youbing Yin, Jingyu Wu, Anoop Kumar, Scott Novotney, Daben Liu, Erin Babinsky
The authors test prompts that help automated judges compare answers by attending to cues such as verbosity, confidence, and authority. Bias-aware prompting flags a cue before judgment, while self-debiasing asks the judge to critique and reconsider its choice. Across three models, self-debiasing improves 20 of 21 model–bias combinations. Gains are measured on factual benchmarks; transfer to subjective tasks remains unvalidated.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.