Reproduce “Information asymmetry in KL-regularized RL” on OpenResearch

About this paper

Alexandre Galashov, Siddhant M. Jayakumar, Leonard Hasenclever, Dhruva Tirumala, Jonathan Schwarz, Guillaume Desjardins, Wojciech M. Czarnecki, Yee Whye Teh and 2 more

The authors introduce a default policy that guides a task-solving policy while receiving less task information. In simulated control, the default sees a walker’s proprioception but not its target, and the task policy is regularized toward it. This improves learning on sparse-reward tasks with complex walkers, while limited or no benefit appears for dense-reward tasks and a simple walker.

Key insights

• Information asymmetry forces the default policy to extract and represent reusable, goal-agnostic behaviors, preventing it from simply mimicking the task-specific agent policy.
• The learned default policy functions as a dynamic prior, injecting beneficial inductive biases that regularize the agent's policy and accelerate learning by encouraging structured exploration.
• The framework's objective can be theoretically connected to the information bottleneck principle and variational EM, providing deeper insights into its mechanism for learning structured behaviors.

Summary by alphaXiv.