Reproduce “OP-Bench: Benchmarking Over-Personalization for Memory-Augmented Personalized Conversational Agents” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yulin Hu, Zimo Long, Jiahe Guo, Xingyu Sui, Xing Fu, Weixiang Zhao, Yanyan Zhao, Bing Qin
The authors introduce OP-Bench to measure over-personalization in memory-augmented dialogue. It uses human-reviewed, synthetic questions grounded in LoCoMo histories to test irrelevance, sycophancy, and repetition, comparing memory-enabled with memory-free answers. For GPT-4o-mini, RAG lowers the macro-averaged score from 83.10 to 55.96, a 32.7% relative drop under the tested setup. This benchmark result does not establish effects on real users.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.