Reproduce “Human Evaluation of Conversations is an Open Problem: comparing the sensitivity of various methods for evaluating dialogue agents” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Eric Michael Smith, Orion Hsu, Rebecca Qian, Stephen Roller, Y-Lan Boureau, Jason Weston

The authors compare methods for evaluating dialogue, testing judgments of individual replies and whole conversations, with either one model rated at a time or two compared side by side. Across three model pairs, the most sensitive method varied: pairwise turn-level ratings led for the fine-tuning preference comparison, with a 93% win rate. The results do not establish a universal rule.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.