Reproduce “Reviving The Classics: Active Reward Modeling in Large Language Model Alignment” on OpenResearch

About this paper

Yunyi Shen, Hao Sun, Jean-François Ton

Adaptation of classical statistical experimental design principles, particularly Fisher information-based D-optimality, for active reward modeling in large language model alignment consistently outperforms existing active learning strategies, reducing the need for extensive human annotation. This method is shown to be computationally efficient and robust across various LLMs and datasets, demonstrating enhanced data efficiency for alignment and benefiting from cross-prompt comparisons.

Key insights

• Classical statistical experimental design principles, such as Fisher Information and D-optimality, can be effectively adapted for modern deep learning-based reward models by applying them to the network's last-layer features.
• Optimal preference comparisons for reward modeling are those that offer both diversity in embedding differences (exploration) and high uncertainty in predicted outcomes (informativeness).
• Cross-prompt comparisons, where responses from different prompts are compared, can significantly enhance the efficiency and stability of human feedback collection for LLM alignment.

Summary by alphaXiv.