Reproduce “Rank-GRPO: Training LLM-based Conversational Recommender Systems with Reinforcement Learning” on OpenResearch

About this paper

Yaochen Zhu, Harald Steck, Dawen Liang, Yinhan He, Vito Ostuni, Jundong Li, Nathan Kallus

ConvRec-R1 is an end-to-end framework for training LLM-based conversational recommender systems, addressing issues like catalog unawareness and degraded ranking quality. It integrates a data-driven supervised fine-tuning stage with a specialized reinforcement learning algorithm, Rank-GRPO, enabling smaller LLMs to match or exceed the performance of larger proprietary models.

Key insights

• A multi-step data distillation pipeline effectively grounds LLMs in specific item catalogs and output structures, serving as a robust warm-start for subsequent reinforcement learning.
• Aligning reinforcement learning policy updates with the rank-structure of outputs, rather than token-level or sequence-level, provides more stable optimization and better credit assignment for generative ranking tasks.
• Smaller open-source LLMs (0.5B-3B parameters), when trained with this structured SFT and rank-aware RL, can achieve performance comparable to or superior to larger, proprietary LLMs for conversational recommendation.

Summary by alphaXiv.