Reproduce “Towards Open-Ended Emotional Support Conversations in LLMs via Reinforcement Learning with Future-Oriented Rewards” on OpenResearch

About this paper

Ting Yang, Li Chen, Huimin Wang

The RLFF-ESC framework optimizes Large Language Models for emotional support conversations, enabling open-ended responses that consider long-term emotional outcomes, and outperforms much larger general-purpose LLMs and traditional methods on success rate and efficiency. This framework leverages multi-agent dialogue simulations to generate future-oriented rewards, facilitating the training of more adaptive and effective support systems.

Key insights

• Multi-agent dialogue simulations can effectively generate scalable, future-oriented feedback for complex conversational tasks, overcoming the limitations of expensive human long-term evaluations.
• Integrating an explicit 'thinking format' reward with a learned future-oriented reward enables LLMs to generate more adaptive, coherent, and empathetic open-ended responses.
• Group Relative Policy Optimization (GRPO) provides a stable and efficient method for fine-tuning LLMs on complex, multi-turn emotional support objectives, outperforming other RL methods like DPO for this task.

Summary by alphaXiv.