Reproduce “Inverse Preference Learning: Preference-based RL without a Reward Function” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Joey Hejna, Dorsa Sadigh

Inverse Preference Learning (IPL) from Stanford University's Autonomous Systems Lab introduces a preference-based Reinforcement Learning method that directly optimizes the Q-function from human preferences, entirely eliminating the need for an explicit reward function. This approach achieves competitive performance while drastically reducing parameter count by over ten times compared to prior methods and improving training stability.

Key insights

  • For a fixed policy, the Q-function inherently contains sufficient information to derive the reward function, making an explicit reward model redundant in preference-based learning.
  • Directly optimizing the Q-function for preferences, rather than an intermediate reward model, simplifies the RL pipeline, reducing architectural complexity and associated hyperparameter tuning.
  • A convex regularization term applied to the implicit reward is crucial to constrain the Q-function optimization, prevent unbounded values, and ensure a unique, stable solution for the implicit reward.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.