Reproduce “Trust Region Reward Optimization and Proximal Inverse Reward Optimization Algorithm” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yang Chen, Menglin Zou, Jiaqi Zhang, Yitan Zhang, Junyi Yang, Gael Gendron, Libo Zhang, Jiamou Liu and 1 more
Researchers from Shanghai Artificial Intelligence Laboratory and the University of Auckland introduce Trust Region Reward Optimization (TRRO), a framework for Inverse Reinforcement Learning (IRL) that provides formal guarantees for monotonically increasing expert likelihood. Their practical algorithm, Proximal Inverse Reward Optimization (PIRO), demonstrates enhanced stability and sample efficiency, achieving superior performance on various MuJoCo and Gym Robotics tasks, and accurately modeling real-world meerkat behavior.
Key insights
- It is possible to establish a formal stability guarantee for non-adversarial IRL by ensuring monotonic improvement in the likelihood of expert behavior, analogous to TRPO/PPO in forward RL.
- Various non-adversarial IRL methods can be viewed under a unified objective: maximizing the likelihood of expert behavior, which directly relates to minimizing the imitation gap.
- An adaptive, L2-regularized approximation of a trust region update can successfully translate theoretical monotonic improvement guarantees into a stable and high-performing practical IRL algorithm for continuous control tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.