Reproduce “Optimizing Novelty of Top-k Recommendations using Large Language Models and Reinforcement Learning” on OpenResearch

About this paper

Amit Sharma, Hua Li, Xue Li, Jian Jiao

Microsoft Research and Bing Ads introduced PG-Ret, a method combining reinforcement learning and large language models to optimize novelty in top-k recommendations. The system achieved 2x-5x gains in novel items across diverse datasets and led to a 1% increase in query-ad matching density during a real-world online A/B test.

Key insights

• Large Language Models can effectively generate synthetic relevance feedback, enabling the training of smaller production models for objectives like novelty without relying on expensive human annotations or real user data.
• Reinforcement Learning can be made scalable for retrieval systems by finetuning a pre-trained base model and reducing the action space to a binary decision, lowering computational burden.
• Optimizing for novelty does not necessarily degrade relevance and can sometimes lead to the discovery of more globally relevant recommendations, improving metrics like precision in certain contexts.

Summary by alphaXiv.