Reproduce “Long-form RewardBench: Evaluating Reward Models for Long-form Generation” on OpenResearch

About this paper

Hui Huang, Yancheng He, Wei Liu, Muyun Yang, Jiaheng Liu, Kehai Chen, Bing Xu, Conghui Zhu and 2 more

This research introduces Long-form RewardBench, the first dedicated benchmark for evaluating reward models (RMs) in long-form text generation across five diverse application scenarios. It evaluates over 20 mainstream RMs, revealing that sequence classifiers generally outperform generative models but struggle with reasoning tasks, while generative models are more sensitive to error position and training data length in long contexts.

Key insights

• Generative reward models exhibit a "lost-in-the-middle" phenomenon, showing lower error detection accuracy when errors are in the middle of long responses, consistent with similar findings in long-context LLMs.
• Sequence classifiers are more robust to variations in response length for error detection but show higher accuracy when errors are situated towards the end of the document, potentially due to training paradigms.
• The impact of training data length differs significantly: sequence classifiers are relatively insensitive, while generative models perform better when trained on longer text groups.

Summary by alphaXiv.