Reproduce “An Empirical Study of Implicit Regularization in Deep Offline RL” on OpenResearch

About this paper

Caglar Gulcehre, Srivatsan Srinivasan, Jakub Sygnowski, Georg Ostrovski, Mehrdad Farajtabar, Matt Hoffman, Razvan Pascanu, Arnaud Doucet

The authors test whether effective rank, a singular-value-based measure of feature directions in an agent’s final hidden layer, tracks offline reinforcement-learning performance. Across Atari comparisons, a positive association in restricted ReLU settings does not persist across broader hyperparameter and model variation. In Asterix, the higher-rank ReLU group had a 0.089 ± 0.025 normalized-score increase; this comparison does not establish causation.

Key insights

• The direct positive correlation between effective rank and agent performance, as suggested by the 'rank collapse hypothesis,' is often spurious and confined to restricted experimental settings, disappearing or reversing under broader conditions.
• Training of deep Q-networks with ReLU activations progresses through three distinct phases: an initial rapid rank collapse ('self-pruning'), a phase of rank increase/plateau with complex behavior learning, and a final collapse often linked to underfitting and dead ReLU units.
• Effective rank dynamics are profoundly influenced by various confounding factors, including activation functions, learning rates, dataset size, and the type of loss function, often independently of agent performance.

Summary by alphaXiv.