Reproduce “P-GenRM: Personalized Generative Reward Model with Test-time User-based Scaling” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Pinyi Zhang, Ting-En Lin, Yuchuan Wu, Jingyang Chen, Zongqi Wang, Hua Yang, Ze Xu, Fei Huang and 2 more

Alibaba Group's P-GenRM introduces a personalized generative reward model for large language models, achieving state-of-the-art performance in aligning LLMs to individual user preferences and significantly improving generalization for new users through a novel test-time user-based scaling mechanism. The model demonstrated a 2.77% average improvement over prior methods on the LLaMA-3.1-8B model, with its 8B variant outperforming existing 70B models by 1.04% on personalized benchmarks.

Key insights

  • Generative reward models can effectively translate diverse and dynamic user preference signals into adaptive, structured evaluation chains, providing greater transparency and flexibility than scalar reward models.
  • Combining multiple individual-level preference inferences with a collaborative filtering-inspired prototype-level scaling at test time significantly enhances performance and generalization, particularly for users with limited data.
  • A multi-stage training approach that integrates supervised learning, reinforcement learning with process and outcome rewards, and curriculum learning is essential for robustly teaching a generative model to personalize evaluations.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.