Reproduce “Role-Playing Evaluation for Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yassine El Boudouri, Walter Nuninger, Julian Alvarez, Yvan Peter
RPEval, a new benchmark for Large Language Models, evaluates role-playing capabilities across emotional understanding, decision-making, moral alignment, and in-character consistency. Findings reveal Gemini-1.5-Pro performs best overall at 62.24%, while GPT-4o struggles significantly with maintaining character consistency, scoring only 5.81% in that dimension.
Key insights
- Gemini-1.5-Pro demonstrated the highest overall role-playing performance at 62.24%, showing balanced capabilities across the evaluated dimensions.
- GPT-4o exhibited a significant weakness in 'in-character consistency,' scoring only 5.81% due to frequent knowledge leakage that broke the assigned persona.
- Smaller models like Llama 3.2 1B showed limitations in complex reasoning tasks such as decision-making and moral alignment but could sometimes outperform larger models in specific aspects of consistency.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.