Reproduce “Self-rationalization improves LLM as a fine-grained judge” on OpenResearch

About this paper

Prapti Trivedi, Aditya Gulati, Oliver Molenschot, Meghana Arakkal Rajeev, Rajkumar Ramamurthy, Keith Stevens, Tanveesh Singh Chaudhery, Jahnavi Jambholkar and 2 more

A self-rationalization methodology, Self-Rationalization (SRE), iteratively refines large language model (LLM) judges by leveraging synthetically generated preference data and Direct Preference Optimization (DPO). This approach significantly enhances the judge models' accuracy and rationale quality for fine-grained evaluation across various benchmarks, outperforming larger and specialized baselines.

Key insights

• Improving the quality of rationales is critical for enhancing an LLM's judgment accuracy and overall calibration, as better reasoning directly leads to better evaluation.
• Direct Preference Optimization (DPO) effectively aligns models with rationales, overcoming potential training signal dilution issues often encountered with long rationales in SFT or RLHF.
• Synthetic preference data, particularly when created with clear margin thresholds from ground truth, is highly effective for driving iterative self-improvement in LLM judge models.
• The combination of an initial SFT foundation followed by iterative DPO proves to be the most effective training strategy for developing high-performing LLM judges.

Summary by alphaXiv.