Reproduce “Recursively Summarizing Books with Human Feedback” on OpenResearch

About this paper

Jeff Wu, Long Ouyang, Daniel M. Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, Paul Christiano

OpenAI researchers developed a method for abstractive summarization of entire novels by combining recursive task decomposition with reinforcement learning from human feedback. This approach enabled models to generate sensible summaries of books hundreds of thousands of words long, with a small percentage matching human quality, and demonstrated that collecting human comparison data is approximately three times faster than collecting demonstrations for such tasks.

Key insights

• The research empirically validates that combining recursive task decomposition with reinforcement learning from human feedback (RLHF) is an effective strategy for training models on tasks beyond direct human supervisory capacity, advancing 'scalable oversight'.
• Reinforcement learning from human comparisons was empirically found to be approximately three times more human-label-efficient than behavioral cloning from demonstrations for collecting data on complex summarization subtasks, especially at scale.
• A single language model trained on hierarchically decomposed summarization tasks can generalize robustly, with models trained on early sections of books performing comparably to those trained on full-tree data for complete book summarization.

Summary by alphaXiv.