Reproduce “Inside the Unfair Judge: A Mechanistic Interpretability Account of LLM-as-Judge Bias” on OpenResearch

About this paper

Zixiang Xu, Sixian Li, Huaxing Liu, Xiang Wang, Shuai Li, Zirui Song, Xiuying Chen

This research develops a mechanistic interpretability framework to analyze and intervene on scoring biases in large language models acting as judges. It identifies bias as an activation-level phenomenon, demonstrating that specific bias types correspond to distinct low-dimensional subspaces within the model's hidden states, which can be causally manipulated to alter judge scores and proactively predict judge degradation with an AUC of 0.82 on unseen benchmarks.

Key insights

• LLM-as-judge bias possesses a stable internal geometric representation within the model's hidden states, where biased inputs are systematically displaced from a baseline activation manifold along type-specific, low-dimensional subspaces that sharpen with depth.
• The identified bias directions are causally load-bearing, allowing for precise, bidirectional control of scoring outcomes through targeted activation steering in hidden states.
• The internal geometric structure of bias is operationally useful, enabling proactive and robust cross-domain prediction of judge degradation instances based on activation features.

Summary by alphaXiv.