Reproduce “Towards Faithful and Controllable Personalization via Critique-Post-Edit Reinforcement Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Chenghao Zhu, Meiling Tao, Tiannan Wang, Dongyi Ding, Yuchen Eleanor Jiang, Wangchunshu Zhou
Researchers from The Chinese University of Hong Kong, Shenzhen, and OPPO introduced a Critique-Post-Edit reinforcement learning framework for personalized large language models, employing a Generative Reward Model that provides textual critiques and multi-dimensional scores. This approach enables LLMs to achieve a deeper understanding of user preferences, mitigating reward hacking, and resulting in models that outperform strong PPO baselines by 11% and surpass GPT-4.1 on personalization benchmarks.
Key insights
- Generative Reward Models (GRMs) with textual critiques are crucial for mitigating reward hacking and providing granular supervision necessary for nuanced personalization tasks.
- An iterative self-correction mechanism, guided by GRM feedback, effectively enables LLMs to develop a 'meta understanding' of user personas, moving beyond superficial factoid insertion.
- Random sampling of edited responses for PPO training surprisingly outperformed reward-based sampling strategies, suggesting the importance of diverse feedback for policy stability, especially when starting from a task-aligned SFT model.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.