Reproduce “Learning Reward and Policy Jointly from Demonstration and Preference Improves Alignment” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Chenliang Li, Siliang Zeng, Zeyi Liao, Jiaxiang Li, Dongyeop Kang, Alfredo Garcia, Mingyi Hong

AIHF jointly learns a policy and reward model from demonstrations and preference pairs, alternating policy optimization against the reward with reward updates using both sources. In an Anthropic-HH test, the 1B Pythia policy scored 1.167 ± 0.157, versus RLHF’s 0.710 ± 0.085, using 10,000 demonstrations and 5,000 preference pairs. These external reward-evaluator scores are not human ratings or universal proof.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.