Reproduce “Think-J: Learning to Think for Generative LLM-as-a-Judge” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Hui Huang, Yancheng He, Hongli Zhou, Rui Zhang, Wei Liu, Weixun Wang, Jiaheng Liu, Wenbo Su

A framework named Think-J enhances generative Large Language Models acting as evaluators by optimizing their "judgment thinking" capabilities. This approach yields judges that provide interpretable rationales alongside accurate preference decisions, with the Think-J-Qwen-2.5-32B model achieving an overall accuracy of 90.5% on RewardBench, surpassing leading proprietary and open-source evaluation models.

Key insights

  • Initializing a judge with a small dataset of high-quality, concisely "clipped" thinking traces from a powerful reasoning model is crucial for establishing effective judgment capabilities.
  • Reinforcement learning, specifically GRPO with rule-based rewards incorporating judgment accuracy, output format, and preference strength, robustly optimizes thinking without extensive human-annotated critiques.
  • Generative LLM judges, when equipped with optimized thinking, can surpass state-of-the-art classifier-based methods in accuracy while providing interpretable rationales, addressing a critical need for transparency in evaluation.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.