Reproduce “GRACE: A Language Model Framework for Explainable Inverse Reinforcement Learning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Silvia Sapora, Devon Hjelm, Alexander Toshev, Omar Attia, Bogdan Mazoure

GRACE (Generating Rewards As CodE), developed at Apple and the University of Oxford, introduces an Inverse Reinforcement Learning framework that infers interpretable, executable Python reward functions directly from expert demonstrations. It achieved competitive policy learning performance on tasks in MuJoCo, BabyAI, and AndroidWorld, often matching ground-truth rewards and outperforming prior methods with an order of magnitude fewer demonstrations, while also fostering the creation of reusable code modules.

Key insights

  • Leveraging LLMs within an evolutionary search paradigm can effectively synthesize high-quality, interpretable reward functions directly from demonstrations without requiring additional explicit task descriptions or ground-truth rewards.
  • Generating rewards as human-readable and executable code inherently provides interpretability and verifiability, addressing a core limitation of black-box deep IRL models.
  • The LLM's strong priors and reasoning capabilities enable superior sample efficiency, allowing accurate reward recovery and policy learning with significantly fewer expert demonstrations than traditional deep IRL methods.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.