Reproduce “Human irrationality: both bad and good for reward inference” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Lawrence Chan, Andrew Critch, Anca Dragan

University of California, Berkeley researchers demonstrated that specific types of human irrationality can, counterintuitively, enhance reward inference when accurately modeled, while unmodeled irrationality significantly degrades performance. Their findings suggest that even approximate models of human biases are critical for robust Inverse Reinforcement Learning.

Key insights

  • Human irrationality, when correctly accounted for, can increase the mutual information between observed behavior and the underlying reward function, making reward inference more effective.
  • Failing to model systematic human irrationalities and instead assuming noisy-rationality leads to dramatically poorer reward inference, potentially worse than relying on prior beliefs alone.
  • Even approximate models of human irrationality (e.g., knowing the type but not exact parameters, or a similar type) provide substantial benefits over standard noisy-rationality assumptions.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.