Reproduce “Reinformed Dreamer: An Asymmetric World Model Efficiently Trained through Latent Guidance” on OpenResearch

About this paper

Gaspard Lambrechts, Adrien Bolland, Daniel Ebi, Damien Ernst

This research introduces Reinformed Dreamer, an asymmetric world model that leverages a novel latent guidance mechanism to efficiently train a single world model for partially observable reinforcement learning. The algorithm provides a tighter variational lower bound for privileged information, leading to improved learning speed and performance across various environments while maintaining significantly higher training efficiency compared to dual-world model approaches.

Key insights

• The Informed Dreamer's 'informed ELBO' is inherently loose due to an 'information gap' where the variational distribution is insufficiently conditioned on the reward and privileged information, impairing representation learning.
• By introducing a privileged encoder directly utilizing next-step privileged information, a significantly tighter lower bound on the log-likelihood of privileged data can be achieved, improving the fidelity of latent representations.
• A latent guidance mechanism, where the privileged encoder indirectly supervises the unprivileged encoder through a shared prior, allows for effective asymmetric learning within a single world model architecture.

Summary by alphaXiv.