Reproduce “DITTO: Offline Imitation Learning with World Models” on OpenResearch

About this paper

Branton DeMoss, Paul Duckworth, Jakob Foerster, Nick Hawes, Ingmar Posner

DITTO (Offline Imitation Learning with World Models) introduces an algorithm that learns expert policies from high-dimensional pixel observations entirely offline by training an agent within a learned world model, leveraging a non-adversarial intrinsic reward based on latent space divergence. The method consistently recovers or exceeds expert performance across all tested Atari environments and a continuous control task, effectively mitigating policy-induced covariate shift without direct environment interaction.

Key insights

• Minimizing on-policy latent divergence within a learned world model is a reliable predictor of generalization performance in the true environment and effectively mitigates covariate shift in offline settings.
• Standard off-policy metrics like expert action prediction accuracy can be negatively correlated with true environment performance, indicating their inadequacy for evaluating generalization in sequential decision-making.
• Leveraging a fixed, non-adversarial intrinsic reward based on latent space similarity offers greater stability and robustness for offline imitation learning compared to complex adversarial objectives.

Summary by alphaXiv.