Reproduce “Imitation Learning from Suboptimal Demonstrations via Meta-Learning An Action Ranker” on OpenResearch

About this paper

Jiangdong Fan, Hongcai He, Paul Weng, Hui Xu, Jie Shao

ILMAR, developed by researchers at the University of Electronic Science and Technology of China and Duke Kunshan University, introduces an imitation learning algorithm that effectively utilizes suboptimal demonstrations by meta-learning an action ranker. The method achieved competitive or superior performance across MuJoCo tasks, particularly when expert data was scarce and suboptimal data was abundant, by weighting demonstrations based on their relative advantage over the current learned policy.

Key insights

• Demonstrations, even if suboptimal, can be valuable if their actions are superior to what the current learned policy would execute, shifting the focus from 'expert' to 'advantageous' actions.
• Meta-learning can effectively guide a discriminator to assign weights that directly optimize for policy-expert alignment, thereby enhancing the utilization of suboptimal data.
• Combining a vanilla loss for initial stability and prior knowledge with a meta loss for refined alignment is crucial for stable and high-performing discriminator training in this context.

Summary by alphaXiv.