Reproduce “RecSys Arena: Pair-wise Recommender System Evaluation with Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhuo Wu, Qinglin Jia, Chuhan Wu, Zhaocheng Du, Shuai Wang, Zan Wang, Zhenhua Dong
Researchers from Tianjin University and Huawei Noah's Ark Lab introduce "RecSys Arena," a framework that leverages Large Language Models for pair-wise comparison of recommender systems. This approach provides human-aligned evaluation, demonstrating strong consistency with traditional offline metrics while offering enhanced discrimination between competitive algorithms across subjective dimensions.
Key insights
- LLMs are more effective in relative (pair-wise) comparison tasks than in absolute scoring for evaluating recommender systems.
- Detailed user profiles and historical interactions, combined with specific evaluation dimensions, enable LLMs to simulate nuanced human judgment.
- The framework bridges the gap between computational metrics and human perception, offering a cost-effective alternative to A/B testing for human-aligned feedback.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.