Reproduce “Evaluating Large Language Models as Generative User Simulators for Conversational Recommendation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Se-eun Yoon, Zhankui He, Jessica Maria Echterhoff, Julian McAuley
Researchers from the University of California, San Diego introduced the first comprehensive, five-task evaluation protocol for Large Language Model (LLM) based user simulators in conversational recommender systems. This protocol identifies systematic deviations of LLM simulators from human behavior, such as popularity bias in item mentions and less diverse recommendation requests, while showing how specific prompting strategies can mitigate some of these issues.
Key insights
- LLM simulators exhibit systematic deviations from human behavior, including a strong popularity bias in item mentions and less diverse, generic recommendation requests.
- Providing interaction history significantly increases the diversity of items mentioned by simulators, while introducing 'pickiness personality' drastically improves preference alignment with human ratings.
- Simulators often provide coherent feedback but can struggle with subtle nuances in requests, sometimes leading to rejections of relevant recommendations, indicating a lack of fine-grained understanding.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.