Reproduce “FACE: A Fine-Grained Reference-Free Evaluator for Conversational Information Access” on OpenResearch

About this paper

Hideaki Joko, Faegheh Hasibi

Researchers at Radboud University developed FACE, a fine-grained and reference-free evaluation method for Conversational Information Access (CIA) systems. It utilizes optimized evaluation instructions and a 'conversation particle' concept to assess dynamic conversational trajectories across multiple quality aspects, demonstrating higher correlation with human judgments and improved interpretability compared to existing LLM-based evaluators.

Key insights

• Decomposing conversations into 'conversation particles' allows for fine-grained, context-aware, and reference-free evaluation that directly addresses the 'Specificity' requirement in conversational evaluation.
• Optimizing and diversifying evaluation instructions using an iterative process on human-human data improves correlation with human judgments and enhances generalizability across unseen systems, different LLM backbones, and diverse conversational domains.
• Leveraging a pool of optimized instructions helps mitigate common LLM evaluation biases (e.g., length bias, self-bias) and reduces the arbitrary nature of prompt engineering, making the evaluation process more robust and efficient.

Summary by alphaXiv.