Reproduce “Synthetic Users, Real Differences: an Evaluation Framework for User Simulation in Multi-Turn Conversations” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yu Lu Liu, Hyokun Yun, Tanya Roosta, Ziang Xiao
The *realsim* framework evaluates the realism of LLM-based user simulations in multi-turn conversations by comparing simulated and real dialogues across eight fine-grained dimensions of user behavior. Its application revealed that current simulated users tend to be "easier" interlocutors, exhibiting fewer errors, less conversational friction, and different task decomposition patterns than real users, which could lead to over-optimistic chatbot performance evaluations.
Key insights
- Simulated users consistently act as "easier" interlocutors, exhibiting less communication friction (e.g., fewer errors, higher readability, more positive feedback) compared to real human users.
- Significant discrepancies exist in how simulated users decompose tasks (Intents) and provide feedback, potentially leading to evaluations that assess different chatbot capabilities than those most relevant for real users.
- The realism of user simulations varies considerably across different conversational domains, highlighting the difficulty of developing a single "general-purpose" simulator that accurately emulates diverse human behaviors.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.