Reproduce “Fast Policy Learning through Imitation and Reinforcement” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ching-An Cheng, Xinyan Yan, Nolan Wagener, Byron Boots
LOKI (Locally Optimal search after K-step Imitation) integrates imitation and reinforcement learning by unifying them under a mirror descent framework. This approach enables policies to quickly approach expert performance and then further improve, ultimately matching or exceeding the performance of idealized RL baselines on continuous robotic control tasks.
Key insights
- First-order RL and IL algorithms can be unified within a common mirror descent optimization framework, differing mainly in their choice of first-order oracle.
- Randomizing the number of imitation steps (K) in LOKI enables theoretical guarantees, ensuring the final policy performs as if running policy gradients directly from the expert.
- Concurrent learning of value function estimates during the imitation phase facilitates seamless and stable transitions to the reinforcement learning phase.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.