Reproduce “Hierarchical Reinforcement Learning for Open-Domain Dialog” on OpenResearch

About this paper

Abdelrhman Saleh, Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Rosalind Picard

Researchers from Harvard and MIT Media Lab developed Variational Hierarchical Reinforcement Learning (VHRL) for open-domain dialogue, allowing models to optimize for human-centered metrics such as reduced toxicity, increased positive sentiment, and active listening. This approach improved human-rated conversation quality and engagement, notably reducing the percentage of swear words in generated utterances by over two-thirds.

Key insights

• Hierarchical Reinforcement Learning, operating at the utterance level, provides more effective credit assignment for long-term conversational goals than word-level RL.
• Joint, end-to-end training of manager and worker policies within VHRL is essential for achieving superior performance in optimizing dialogue quality.
• Directly optimizing for human-centered reward functions can effectively mitigate issues like toxicity and repetitiveness while promoting positive conversational attributes.

Summary by alphaXiv.