Reproduce “Delayed Reinforcement Learning by Imitation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Pierre Liotet, Davide Maran, Lorenzo Bisi, Marcello Restelli
This research from Politecnico di Milano presents Delayed Imitation with Dataset Aggregation (DIDA), an algorithm that addresses constant, non-anonymous delays in reinforcement learning by training an agent to imitate an expert policy learned in the undelayed environment. DIDA demonstrates superior sample efficiency and high performance across various continuous control and real-world tasks, including robotic locomotion and financial trading, while also exhibiting robustness to increased delays.
Key insights
- The 'duality of trajectories' allows for the application of Dataset Aggregation (DAGGER) in delayed environments by constructing augmented states from delayed observations and querying an undelayed expert for the true current state.
- Theoretical analysis demonstrates that the performance difference between an undelayed expert and a perfectly imitated delayed policy is related to the variance of the belief distribution over the current state and proportional to the delay under time-Lipschitz conditions, suggesting that smoother expert policies are beneficial.
- Imitation learning can provide a regularization effect, leading the delayed policy to sometimes outperform the expert on unseen data, as observed in the trading application.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.