Reproduce “Robust Reinforcement Learning using Offline Data” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Kishan Panaganti, Zaiyan Xu, Dileep Kalathil, Mohammad Ghavamzadeh
Proposing Robust Fitted Q-Iteration (RFQI), this work presents the first algorithm to address offline robust reinforcement learning for arbitrarily large state spaces using function approximation, complete with provable performance guarantees. RFQI empirically demonstrates superior robust performance compared to non-robust and soft-robust RL algorithms on standard benchmarks, maintaining performance under various model perturbations.
Key insights
- The robust Bellman operator can be dually reformulated to exclusively depend on the nominal data distribution, making offline robust reinforcement learning tractable.
- Optimal dual variables for robust Bellman updates can be approximated through functional Empirical Risk Minimization over a function class, which enables the algorithm to scale to large state and action spaces using function approximation.
- The proposed Robust Fitted Q-Iteration (RFQI) algorithm provides non-asymptotic, approximately optimal performance guarantees, including a sample complexity of O(1/√N) for learning robust policies from offline data.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.