Reproduce “PersonaChatBench: Evaluating Chat Assistants with Diverse Simulated Users” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zheyuan Deng, Taoran Li, Jingwen Zhang, Yuante Li, Dachuan Zhao, Fulei Zhang, Yunkai Zhang, Zhikun Xu and 23 more
PersonaChatBench tests same tasks with simulated users across 806 tasks, six domains, and 20 language models. It separately measures task performance, fulfillment of expressed needs, and simulator fidelity. In math tutoring, hint-first preferences scored 75.3 versus 86.6 for demonstration-first preferences; an 11.3-point gap remained when all six conditions reached correct conclusions. These rubric-based scores measure expressed needs, not human satisfaction.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.