Reproduce “PaTaRM: Bridging Pairwise and Pointwise Signals via Preference-Aware Task-Adaptive Reward Modeling” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ai Jian, Jingqing Ruan, Xing Ma, Xiaoyun Zhang, Dailin Li, Weipeng Zhang, Ke Zeng, Xunliang Cai
PaTaRM introduces a generative reward model that enables pointwise training directly from readily available pairwise preference data, eliminating the need for expensive absolute annotations. The framework enhances interpretability through dynamic, context-aware rubrics and demonstrates improved generalization on reward benchmarks and substantial gains in downstream RLHF policy performance across instruction-following and reasoning tasks, exhibiting robustness to noisy labels.
Key insights
- Training a generative reward model to perform pointwise evaluations directly from pairwise preference data is effective, providing an O(N) inference complexity without requiring explicit absolute score annotations.
- Dynamic rubric adaptation, which blends primary task criteria with model-generated constraints, balances stability and adaptability, leading to more precise and context-aware evaluations that mitigate reward hacking.
- The Preference-Aware Reward (PAR) mechanism's stochastic score aggregation and dense rollout-level credit assignment implicitly act as a multi-sample voting mechanism, providing resilience to noisy labels and improving training stability.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.