Reproduce “References Improve LLM Alignment in Non-Verifiable Domains” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Kejian Shi, Yixin Liu, Peifeng Wang, Alexander R. Fabbri, Shafiq Joty, Arman Cohan

The research introduces "RefEval", a method leveraging high-quality reference outputs to guide large language model (LLM) evaluators, effectively acting as soft verifiers in non-verifiable domains. This approach enhances LLM evaluation accuracy by up to 17.4% for smaller models and enables self-improvement through Direct Preference Optimization (DPO) that matches or exceeds models trained with external reward models, without costly human or AI preference data.

Key insights

  • Reference-guided prompting significantly improves LLM-as-a-Judge evaluation accuracy and inter-judge agreement, providing greater benefits for less capable LLM judges by offering objective grounding.
  • High-quality references can effectively serve as "soft verifiers" for LLM alignment in domains traditionally considered non-verifiable, bridging a methodological gap between RLVR and RLHF/RLAIF paradigms.
  • LLMs can effectively self-supervise their alignment training using reference-guided self-judges, achieving performance comparable to or exceeding methods that rely on external, separately finetuned reward models without their associated data costs.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.