Reproduce “Generalization in offline RL: The structure is more important than the amount of pessimism” on OpenResearch

About this paper

Max Weltevrede, Matthijs T.J. Spaan, Wendelin Böhmer

Researchers at Delft University of Technology investigated the role of pessimism in offline reinforcement learning generalization, finding that the structural alignment of pessimism with environmental symmetries is more crucial than its magnitude for zero-shot policy transfer. Their work introduces Data Augmentation Consistency (DAC) regularization, particularly DAC-Output, which achieved near-optimal generalization performance (e.g., 0.98 with expert data) in a rotationally symmetric environment.

Key insights

• Theoretical analysis (Theorem 1) demonstrated that if a pessimistic Q-value function is symmetric, its magnitude does not prevent optimal zero-shot generalization to unseen contexts.
• Conversely (Theorem 2), a milder pessimistic value function that violates underlying symmetries can lead to arbitrarily worse generalization compared to an aggressively pessimistic but symmetric one.
• Data Augmentation Consistency (DAC), particularly applied to the network output during policy extraction, effectively enforces structural symmetries and significantly improves generalization performance.

Summary by alphaXiv.