Reproduce “Path Channels and Plan Extension Kernels: a Mechanistic Description of Planning in a Sokoban RNN” on OpenResearch

About this paper

Mohammad Taufeeque, Aaron David Tucker, Adam Gleave, Adrià Garriga-Alonso

An investigation into a model-free reinforcement learning agent for Sokoban mechanistically elucidated its planning process, identifying internal "path channels" that represent future movements and a bidirectional planning algorithm built from specialized convolutional kernels. This work revealed mechanisms for plan extension, stopping, backtracking, and a Winner-Takes-All selection system, which consistently emerged across multiple trained networks.

Key insights

• Model-free reinforcement learning agents can implicitly learn sophisticated planning algorithms, including bidirectional search, through specialized internal circuits and recurrent computations.
• The network develops an internal, value-like representation within path channels that implicitly prioritizes path length, suggesting characteristics of a learned "mesa-optimizer."
• Detailed, weight-level mechanistic interpretability for complex, high-performing neural networks is feasible, moving beyond probe-based or localized feature analyses.

Summary by alphaXiv.