Reproduce “Eliminating Inductive Bias in Reward Models with Information-Theoretic Guidance” on OpenResearch

About this paper

Zhuo Li, Pengyu Cheng, Zhechao Yu, Feifei Tong, Anningzhe Gao, Tsung-Hui Chang, Xiang Wan, Erchao Zhao and 2 more

Researchers from Alibaba and The Chinese University of Hong Kong, Shenzhen, developed Debiasing via Information optimization for RM (DIR), an information-theoretic method to eliminate inductive biases in reward models used for LLM alignment. This approach significantly improved reward model robustness and led to LLMs with enhanced performance across various benchmarks, exhibiting reduced length, sycophancy, and format biases.

Key insights

• Formulating the RM debiasing problem within an information-theoretic framework provides a unified and theoretically grounded approach to handle diverse and complex non-linear inductive biases.
• Explicitly minimizing mutual information between reward model representations and specific bias attributes effectively disentangles genuine human preferences from spurious correlations present in training data.
• The use of relative bias attributes and difference-based hidden state representations enhances the efficacy of debiasing, making the method robust and compatible with other alignment techniques like DPO.

Summary by alphaXiv.