Reproduce “Large Language Models as User-Agents for Evaluating Task-Oriented-Dialogue Systems” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Taaha Kazi, Ruiliang Lyu, Sizhe Zhou, Dilek Hakkani-Tur, Gokhan Tur
This paper develops an automated, comprehensive evaluation framework for Task-Oriented Dialogue (TOD) systems by leveraging Large Language Models (LLMs) as both dynamic user simulators and reliable evaluator agents. The framework integrates advanced prompting strategies to model complex user behaviors and demonstrates high agreement between LLM-based and human evaluations for task completion, particularly with GPT-4.
Key insights
- LLMs, specifically GPT-4, can serve as reliable automated evaluators for Task-Oriented Dialogue (TOD) systems, demonstrating substantial agreement (Cohen's Kappa 0.796, 90.0% accuracy) with human judgments for task completion.
- Different prompting strategies for LLM user agents lead to varied conversational behaviors; the Vanilla prompt often achieves higher keyword fulfillment, while the User State Tracking prompt enables more sophisticated interactions like negotiation and goal adaptation.
- The choice of LLM for the TOD system and the user simulator's prompting strategy significantly impacts performance metrics, with Llama 3 consistently outperforming other TOD models in task completion within this framework.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.