Reproduce “Towards a Design Guideline for RPA Evaluation: A Survey of Large Language Model-Based Role-Playing Agents” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Chaoran Chen, Bingsheng Yao, Ruishi Zou, Wenyue Hua, Weimin Lyu, Yanfang Ye, Toby Jia-Jun Li, Dakuo Wang
Researchers from institutions including the University of Notre Dame and Northeastern University conducted a systematic review of LLM-based role-playing agents (RPAs) to develop an evidence-based, two-step guideline for their evaluation. This framework categorizes agent and task attributes and links them to appropriate metrics, demonstrating how it can enhance evaluation consistency and comprehensiveness.
Key insights
- Six distinct categories of agent attributes (e.g., activity history, psychological traits) and seven task attribute categories (e.g., simulated individuals, opinion dynamics) define current RPA designs and applications.
- Seven common evaluation metric categories (e.g., performance, psychological, external alignment) are employed, but their selection often lacks systematic justification.
- Specific patterns reveal that certain agent and task attributes are consistently evaluated with particular metric types, forming the basis for a structured evaluation guideline.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.