Reproduce “Sliding Puzzles Gym: A Scalable Benchmark for State Representation in Visual Reinforcement Learning” on OpenResearch

About this paper

Bryan L. M. de Oliveira, Luana G. B. Martins, Bruno Brandão, Murilo L. da Luz, Telma W. de L. Soares, Luckeciano C. Melo

Researchers introduced the Sliding Puzzles Gym (SPGym), a benchmark designed to isolate and systematically evaluate visual representation learning in reinforcement learning by scaling visual diversity while keeping task dynamics fixed. Empirical evaluation with SPGym revealed that current state-of-the-art visual RL algorithms struggle significantly with generalization to unseen visual inputs and exhibit performance degradation as visual diversity increases.

Key insights

• SPGym effectively differentiates the performance of various visual RL algorithms, showing that approaches like DreamerV3 and data augmentation (RAD) are more robust to increasing visual diversity than other self-supervised or reconstruction-based methods.
• A critical finding is the universal failure of evaluated agents to generalize to completely unseen images (Hard OOD), indicating a reliance on memorization of training visuals rather than acquiring generalizable visual understanding.
• Increasing training image diversity did not improve generalization to completely novel images and often decreased performance on even slightly augmented versions of training images, suggesting limitations in current methods' capacity for broad visual generalization.

Summary by alphaXiv.