Reproduce “CORA: Benchmarks, Baselines, and Metrics as a Platform for Continual Reinforcement Learning Agents” on OpenResearch

About this paper

Sam Powers, Eliot Xing, Eric Kolve, Roozbeh Mottaghi, Abhinav Gupta

The CORA platform establishes a unified, open-source framework for Continual Reinforcement Learning, offering diverse benchmarks, standardized metrics, and re-implemented baselines. This initiative significantly reduces compute costs for experiments, with new benchmarks running up to seven times faster, while also revealing considerable limitations of current CRL methods in generalization and long-sequence plasticity.

Key insights

• The new benchmarks (Procgen, MiniHack, CHORES) offer substantial computational efficiency improvements, with MiniHack running 7x faster than prior Atari experiments.
• Current state-of-the-art CRL methods exhibit notable weaknesses in out-of-distribution generalization, maintaining plasticity over extended task sequences, and achieving sample efficiency in complex environments like CHORES.
• Replay-based algorithms, such as CLEAR, generally outperform other baselines in mitigating catastrophic forgetting but still struggle significantly with long-sequence learning and transfer in challenging domains.

Summary by alphaXiv.