Reproduce “Exploring Diversity, Novelty, and Popularity Bias in ChatGPT's Recommendations” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Dario Di Palma, Giovanni Maria Biancofiore, Vito Walter Anelli, Fedelucio Narducci, Tommaso Di Noia
This study evaluates ChatGPT models (GPT-3.5 and GPT-4) as recommenders, moving beyond accuracy to quantify their performance on diversity, novelty, and popularity bias across three datasets. It finds that GPT-4 effectively balances these factors and proves particularly robust in challenging user cold-start scenarios.
Key insights
- Role-Playing prompting proved the most effective strategy for eliciting consistent and high-quality recommendations from ChatGPT, outperforming Zero-Shot, Few-Shot, and Chain-of-Thought methods.
- ChatGPT models, particularly GPT-4, demonstrate a strong capability to balance diversity and novelty in recommendations, often outperforming many traditional collaborative and content-based filtering baselines.
- ChatGPT exhibits a robust ability to provide relevant, diverse, and novel recommendations even in user cold-start scenarios, a challenging problem for traditional recommender systems.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.