Reproduce “On the Feasibility of Learning, Rather than Assuming, Human Biases for Reward Inference” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Rohin Shah, Noah Gundotra, Pieter Abbeel, Anca D. Dragan

Researchers at UC Berkeley explore whether learning human planning biases directly from data can improve reward inference for AI systems compared to assuming fixed human rationality models. Their work shows that while learning biases offers advantages over incorrect assumptions, the approximation error of current differentiable planners, like Value Iteration Networks, presents the most significant limitation to practical performance.

Key insights

  • Learning human biases through a differentiable planner generally improves reward inference compared to assuming fixed (and potentially incorrect) models of human rationality (optimal or Boltzmann).
  • The most significant barrier to the proposed learning approach is the approximation error inherent in current differentiable planning architectures, such as Value Iteration Networks (VINs).
  • For scenarios without known rewards, initializing the learned planner to mimic an optimal agent is crucial, serving as an effective regularization strategy to make the learning problem tractable.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.