Reproduce “CIG: Exploration via Conditional Information Gain” on OpenResearch

About this paper

Tim Joseph, Marcus Fechner, Philipp Stegmaier, Karam Daaboul, J. Marius Zöllner

CIG changes reward-free exploration in model-based reinforcement learning by scoring imagined steps for uncertainty not already covered earlier in a rollout. It uses disagreement among learned predictors, with a prefix correction that discounts repeated probes. Across twelve DreamerV2 tasks, CIG leads normalized scores and beats each baseline with probability above 0.79 on a sampled task, but not every task.

Key insights

• Approximating trajectory-level information gain with the entropy of a moment-matched Gaussian mixture is feasible, but naive implementation suffers from rank saturation, limiting its utility for multi-step exploration in deep models.
• Collapsing the epistemic covariance blocks to their trace significantly increases the effective information capacity, overcoming the rank-saturation issue and enabling sustained exploration signals over imagined rollouts.
• The Cholesky decomposition of the modified kernel naturally yields per-step rewards that filter irreducible noise via an aleatoric ridge, provide lifetime uncertainty from ensemble disagreement, and penalize within-rollout redundancy through prefix correction.

Summary by alphaXiv.