Reproduce “LaMP-QA: A Benchmark for Personalized Long-form Question Answering” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Alireza Salemi, Hamed Zamani
LaMP-QA is introduced as the first benchmark specifically for personalized long-form question answering, addressing a gap in evaluating tailored answer generation for information-seeking. The benchmark demonstrates that incorporating user profiles can improve performance by up to 39% and validates an aspect-based evaluation method highly aligned with human judgment.
Key insights
- Personalization leveraging user profiles provides substantial performance improvements in long-form answer generation, with up to 39% gain over non-personalized approaches.
- Incorporating an explicit planning step, as demonstrated by Plan-RAG Personalization, enhances personalized answer generation by guiding the LLM with inferred user-specific aspects.
- The proposed aspect-based LLM evaluation method offers a reliable and fine-grained approach to assess personalized responses, showing 73% alignment with human judgment.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.