Reproduce “Imitating careful experts to avoid catastrophic events” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jack R. P. Hanslope, Laurence Aitchison
The authors use carefulness as an additional signal for inferring rewards through inverse reinforcement learning (IRL). In a toy gridworld, carefulness makes movement less stochastic at an increased cost. The method inferred cliff-state rewards of −960 to −1000, compared with −7 to −11 for a baseline, against a true value of −1000; one human expert’s result varied substantially.
Key insights
- "Carefulness" signals, whether explicitly defined or implicitly observed in expert demonstrations, contain crucial information that resolves the underdetermination problem in IRL for catastrophic events.
- Experts naturally exhibit increased carefulness (e.g., slowing down, increased precision) when nearing potential catastrophic outcomes, providing an observable and learnable cue for reward severity.
- Leveraging carefulness enables IRL algorithms to learn severe negative rewards with significantly fewer expert demonstrations compared to methods that do not use such signals, improving data efficiency for safety-critical learning.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.