Reproduce “Reward-free Pretraining for Reinforcement Learning via Occupancy Coverage Maximization” on OpenResearch

About this paper

Marco Pratticò, Pietro Novelli, Massimiliano Pontil, Carlo Ciliberto

ROVER trains a reward-free exploration policy to visit diverse reachable states despite unknown future goals. It learns representations and a transition model, estimates discounted occupancy, and reduces its concentration, routing unsupported predictions to a sink state. In custom navigation tasks, its final policy needs about tenfold fewer observations for full coverage than exploration from scratch; tests use simple environments with discrete actions.

Key insights

• Minimizing the squared RKHS norm of the discounted state occupancy's mean embedding directly corresponds to maximizing the order-2 Rényi entropy, providing a principled, target-free objective for promoting broad and uniform state-space coverage.
• The sink-state augmentation acts as an implicit regularization mechanism, effectively assigning a cost to venturing into unobserved or unsupported state-action regions, which stabilizes exploration and encourages the policy to retain coverage of known states.
• Leveraging operator world models allows for closed-form and differentiable computation of the discounted state occupancy objective's functional gradient, enabling efficient policy updates without direct access to reward signals.

Summary by alphaXiv.