Reproduce “Visual Transfer for Reinforcement Learning via Wasserstein Domain Confusion” on OpenResearch

About this paper

Josh Roy, George Konidaris

WAPPO, an algorithm developed at Brown University, enhances zero-shot visual transfer for Deep Reinforcement Learning agents by learning domain-agnostic feature representations. It achieves this by explicitly minimizing the Wasserstein-1 distance between feature distributions from source and target environments using an adversarial training framework, outperforming prior methods on 14 out of 16 easy and 12 out of 16 hard OpenAI Procgen tasks.

Key insights

• The use of Wasserstein-1 distance (Earth Mover's Distance) in an adversarial setup provides greater stability and effectiveness in aligning arbitrary feature distributions compared to other divergence measures.
• For Block MDPs, an optimal policy depends only on the hidden state, motivating the learning of representations that are invariant to visual domain shifts while preserving task-relevant information.
• The approach achieves zero-shot transfer without requiring additional computation during inference, making it practical for deployment, and operates without assuming access to paired observations across domains.

Summary by alphaXiv.