Reproduce “LongLaMP: A Benchmark for Personalized Long-form Text Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ishita Kumar, Snigdha Viswanathan, Sushrita Yerra, Alireza Salemi, Ryan A. Rossi, Franck Dernoncourt, Hanieh Deilamsalehy, Xiang Chen and 6 more
Researchers from the University of Massachusetts Amherst, Adobe Research, and the University of Oregon introduced LongLaMP, an open-source benchmark designed for personalized long-form text generation across four diverse real-world tasks. Evaluations on this benchmark demonstrate that integrating personalization via Retrieval-Augmented Generation (RAG) yields substantial performance improvements, achieving average gains of 30.21% in ROUGE-1 and 47.5% in ROUGE-L over non-personalized baselines.
Key insights
- Personalization consistently and substantially improves the quality of long-form text generation across various tasks, models (e.g., LLaMA-2, GPT-3.5, FLAN-T5), and evaluation settings.
- The Retrieval-Augmented Generation (RAG) framework offers an effective, scalable, and computationally efficient method for integrating user-specific personalization into LLMs without extensive per-user fine-tuning.
- The optimal number of retrieved user profiles (`k`) and the choice of retriever (lexical vs. dense) are task-dependent, indicating that adaptive strategies for context integration are crucial for best performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.