Reproduce “User Preference Induction with LLMs for Offline Top-N Recommendation Evaluation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
David Otero, Javier Parapar
Researchers at the Universidade da Coruña introduced a framework leveraging Large Language Models to generate personalized relevance judgments for offline top-N recommender system evaluation, addressing incomplete feedback. This approach, particularly when using Induced Preference Profiles (IPP), reduced popularity bias by up to 20% on the MovieLens 32M dataset but amplified it on the head-skewed Goodbooks-10k dataset.
Key insights
- LLM judgments demonstrate strong reliability at the system-ranking level (Kendall's "u03c4" up to 1.0 for NDCG@100), despite some label-level discrepancies, making them suitable for evaluation.
- Induced Preference Profiles (IPP), which provide abstract summaries of user tastes, are crucial for mitigating popularity bias by guiding the LLM judge beyond popularity-laden item mentions in raw profiles.
- The impact of LLM-based relevance judgment on popularity bias is conditional: it reduces bias in datasets with meaningful long tails but can amplify it in inherently head-skewed catalogs.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.