Reproduce “Quantile Regression for Distributional Reward Models in RLHF” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Nicolai Dorka
Quantile Reward Models (QRMs) are introduced, which learn a distribution over human preferences for Reinforcement Learning from Human Feedback (RLHF) instead of a single scalar value. This approach improves reward model quality on benchmarks, achieving 91.2 with a Llama-3 8B backbone, and enables the training of risk-aware LLM policies that reduce undesirable responses without sacrificing overall performance.
Key insights
- Human preferences, often diverse and conflicting, are more accurately represented as a distribution rather than a single point estimate.
- Quantile regression, successful in distributional reinforcement learning, can be effectively adapted to model the spread and uncertainty of human preferences in RLHF reward modeling.
- Leveraging distributional reward information allows for direct control over LLM policy behavior, enabling risk-averse optimization to avoid negative outcomes without compromising overall performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.