Reproduce “RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zekun Moore Wang, Zhongyuan Peng, Haoran Que, Jiaheng Liu, Wangchunshu Zhou, Yuhan Wu, Hongcheng Guo, Ruitong Gan and 9 more
RoleLLM introduces a framework and the RoleBench dataset to benchmark and enhance the character-level role-playing abilities of large language models. The fine-tuned open-source models, RoleLLaMA and RoleGLM, demonstrate strong performance, often matching or exceeding proprietary models like GPT-4 in specific aspects such as role-specific knowledge recall.
Key insights
- Fine-tuning open-source LLMs with high-quality, role-specific data can embed complex personas directly into model weights.
- System-instruction-based conditioning (RoCIT) is a context-efficient and effective method for enabling role-playing in fine-tuned models, often outperforming retrieval augmentation.
- The 'question-confidence-answer' triplet generation process significantly improves data quality and factualness for knowledge injection into role profiles.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.