Reproduce “USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Shikib Mehri, Maxine Eskenazi

Researchers at Carnegie Mellon University developed USR, an UnSupervised and Reference-free evaluation metric for dialog generation, leveraging fine-tuned pre-trained language models. USR measures multiple dialog qualities and demonstrates substantially stronger correlation with human judgments compared to existing reference-based metrics, particularly achieving a Spearman correlation of 1.0 at the system level on two distinct dialog datasets.

Key insights

  • Human perception of dialog quality is multi-faceted, encompassing attributes like understandability, naturalness, context maintenance, interestingness, and knowledge use, which can be individually measured.
  • Fine-tuning large pre-trained language models on raw dialog data, without explicit quality labels, can effectively capture nuanced aspects of dialog quality.
  • Combining diverse, reference-free sub-metrics provides a more robust and interpretable evaluation of dialog quality than single-score, reference-based methods.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.