Reproduce “Robust Preference Learning for Storytelling via Contrastive Reinforcement Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Louis Castricato, Alexander Havrilla, Shahbuland Matiana, Michael Pieler, Anbang Ye, Ian Yang, Spencer Frazier, Mark Riedl
Researchers from CarperAI and Georgia Tech developed CARP CoOp, a contrastive reinforcement learning framework, that enables a 750M parameter language model to generate stories significantly aligned with complex human preferences. This approach achieved an average human preference selection of 61.5% for topic labels and 67.5% for character alignment, outperforming a 20B parameter model in human evaluations.
Key insights
- A robust preference model built via contrastive learning can effectively serve as a reward function for reinforcement learning to guide generative LLMs toward complex human preferences.
- Integrating pseudo-labeling and prompt tuning into the preference model (CARP CoOp) creates a stronger, more discriminative, and stable reward signal, overcoming issues of reward collapse in RL for text generation.
- Effective alignment with human preferences can enable a significantly smaller generative model (GPT-2-750M) to outperform a much larger one (GPT-NeoX-20B) for subjective storytelling tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.