Reproduce “Fighting Copycat Agents in Behavioral Cloning from Observation Histories” on OpenResearch

About this paper

Chuan Wen, Jierui Lin, Trevor Darrell, Dinesh Jayaraman, Yang Gao

Researchers from Tsinghua University, UC Berkeley, and the University of Pennsylvania identified the "copycat problem" in imitation learning from observation histories, where agents learn to repeat previous actions rather than map observations to actions. They developed an adversarial learning approach using a Target-Conditioned Adversary and an Information Bottleneck, which successfully mitigated this issue and improved agent performance across six robotic control tasks.

Key insights

• The "copycat problem" is a specific form of causal confusion in imitation learning from observation histories, where agents learn to predict the next action by simply repeating the previous one due to strong temporal action correlations.
• Higher "action predictability" (the degree to which an agent's next action can be predicted from its previous action) is inversely correlated with actual task performance in imitation learning.
• Actively suppressing information about the previous action from the learned feature representation, combined with an information bottleneck, can effectively mitigate the copycat problem.

Summary by alphaXiv.