Reproduce “CausalRM: Causal-Theoretic Reward Modeling for RLHF from Observational User Feedbacks” on OpenResearch

About this paper

Hao Wang, Licheng Pan, Zhichao Chen, Chunyuan Zheng, Zhixuan Chu, Xiaoxi Li, Yuan Lu, Xinggao Liu and 2 more

CausalRM introduces a causal-theoretic framework to train reward models for large language models (LLMs) using observational user feedback, effectively addressing both user annotation errors and user preference bias. This approach leads to more accurate reward signals, demonstrated by consistently higher performance metrics ($R^2$ of 0.283 on HelpSteer) and improved downstream LLM safety alignment, achieving a 49.2% gain in safety score on WildGuardMix for Qwen2.5-7B.

Key insights

• Both user annotation errors and user preference bias must be jointly addressed in observational reward modeling to accurately estimate true user preferences.
• A causal-theoretic approach can rigorously derive unbiased estimators for the ideal reward modeling objective from noisy and biased observational data.
• The Doubly Robust estimator offers reduced variance and increased robustness against misspecification of nuisance models compared to the IPS approach, making it more practical for real-world observational data.

Summary by alphaXiv.