Reproduce “PRMB: Benchmarking Reward Models in Long-Horizon CBT-based Counseling Dialogue” on OpenResearch

About this paper

Yougen Zhou, Qin Chen, Ningning Zhou, Jie Zhou, Liang He

A new benchmark, PRMB, evaluates reward models for long-horizon, multi-session cognitive behavioral therapy (CBT) counseling dialogues, revealing that current models struggle with consistency and robustness in complex therapeutic contexts. This work provides a specialized tool for assessing how well reward models align with therapeutic principles over extended interactions.

Key insights

• Existing reward models, both discriminative and generative, demonstrate significant limitations in consistency and robustness when evaluated in long-horizon CBT counseling scenarios.
• Performance degrades substantially from pairwise to Best-of-N evaluations, and in later therapeutic sessions, indicating difficulty with nuanced, cumulative therapeutic interactions.
• Retrieval-Augmented Generation (RAG) consistently enhanced generative reward model performance, while other inference strategies like few-shot prompting, Self-Refine, and Chain-of-Thought proved detrimental.

Summary by alphaXiv.