Reproduce “Evaluating Language Models in Realistic Conversational Contexts” on OpenResearch

About this paper

Ilija Subasic, Andrew Rabinovich, Zhao Chen

Researchers from Upwork Inc. developed UPHELD, a large, human-annotated benchmark of multi-turn dialogues for evaluating conversational AI, and proposed a "Mixture-of-Judges" framework. This ensemble approach improved correlation with human conversational quality assessments by 30-40% compared to individual automatic or LLM-as-a-judge metrics.

Key insights

• Traditional automatic metrics and single LLM-as-a-judge methods show only weak to moderate correlations with human judgments for conversational quality, particularly for stylistic nuances, failing to capture the complexity of human evaluation.
• The UPHELD dataset contains learnable behavioral patterns and nuances, as demonstrated by a fine-tuned model (Uma-V0.5) showing a 40% improvement in conversational fidelity over its base model, matching or surpassing newer state-of-the-art LLMs.
• Ensemble metrics, combining multiple evaluative signals, significantly outperform individual metrics and exhibit robust transferability to external datasets, suggesting a more reliable approach to conversational LLM evaluation.

Summary by alphaXiv.