Reproduce “LLM-Eval: Unified Multi-Dimensional Automatic Evaluation for Open-Domain Conversations with Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yen-Ting Lin, Yun-Nung Chen
The LLM-EVAL framework presents a unified multi-dimensional automatic evaluation method for open-domain conversations, utilizing a single large language model prompt and call. This approach achieves higher correlation with human judgments compared to prior state-of-the-art metrics across diverse datasets and evaluation dimensions, streamlining the assessment of dialogue systems.
Key insights
- The efficiency of using a single LLM prompt and call for multi-dimensional evaluation is highly effective, often surpassing more complex or traditional methods.
- LLM-EVAL consistently achieves high correlation with human judgments across various conversation quality aspects and in both reference-based and reference-free scenarios.
- The choice of the underlying LLM is critical, with dialogue-optimized models and greedy decoding strategies yielding better evaluation performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.