Reproduce “Building Legal Reward Models for Grounding and Abstention” on OpenResearch

About this paper

Rilton Franzone, Valentin Noël, Puyu Wang, Philip Torr, Fabio J. Fehr

The authors introduce LegalRewardBench, a benchmark for evaluating whether reward models prefer legal answers grounded in retrieved documents, including when they should abstain. They create contextual preference pairs from legal question-answering data and find that length-balanced DPO raises Ministral-8B pairwise accuracy to 84.9%, from 59.3%, on 280 pairs. This measures pairwise judgments, not end-to-end legal RAG performance.

Key insights

• Grounding behavior transfers from general contextual preference data: training on ContextualJudgeBench alone improves performance on the legal benchmark more than training only on the original legal pairs.
• Hard semantic and topical distractors make both evidence use and abstention substantially more difficult than random retrieval noise.
• Contrastive DPO is more effective than matched supervised fine-tuning, indicating that learning the preference margin between grounded and rejected responses is important.

Summary by alphaXiv.