Reproduce “Limitations of Current Evaluation Practices for Conversational Recommender Systems and the Potential of User Simulation” on OpenResearch

About this paper

Nolwenn Bernard, Krisztian Balog

Researchers investigated the limitations of current evaluation practices for conversational recommender systems (CRSs), proposing new user-centric utility metrics and exploring user simulation for dynamic evaluation. Their Reward-per-Dialogue-Length (RDL) metric correlated with self-reported user satisfaction at Kendall's "u03c4" = 0.78, and LLM-based user simulators demonstrated better performance approximation than agenda-based ones.

Key insights

• The proposed Reward-per-Dialogue-Length (RDL) metric shows a strong alignment with user satisfaction, unlike traditional metrics such as Recall@N, which had minimal correlation.
• Current conversational recommender systems generally exhibit low user satisfaction and RDL scores, highlighting substantial room for improvement in their capabilities.
• LLM-based user simulators demonstrate a greater capacity to approximate real user performance and relative CRS rankings compared to agenda-based simulators, suggesting their potential for automated evaluation.

Summary by alphaXiv.