Reproduce “Stable Hadamard Memory: Revitalizing Memory-Augmented Agents for Reinforcement Learning” on OpenResearch

About this paper

Hung Le, Kien Do, Dung Nguyen, Sunil Gupta, Svetha Venkatesh

This work introduces Stable Hadamard Memory (SHM), a new memory model designed for reinforcement learning agents operating in partially observable environments, addressing challenges of instability and inefficiency in existing memory-augmented networks. SHM consistently surpasses state-of-the-art memory models across meta-RL, long-horizon credit assignment, and POPGym benchmarks, achieving near-optimal success rates and up to perfect scores in certain tasks while maintaining competitive inference speeds.

Key insights

• Proper memory calibration, involving dynamic, element-wise adjustment of memory values, is crucial for effective information management and preventing numerical instabilities in deep-learning memory models.
• Bounding calibration matrix elements using the hyperbolic tangent function, coupled with randomized parameter selection, can theoretically stabilize cumulative products of calibration matrices and improve gradient flow over long sequences.
• The Hadamard product provides an efficient, element-wise memory update mechanism that allows for dynamic and selective forgetting or reinforcement of information without content mixing, critical for adaptation in complex environments.

Summary by alphaXiv.