Reproduce “Decision Mamba: Reinforcement Learning via Sequence Modeling with Selective State Spaces” on OpenResearch

About this paper

Toshihiro Ota

Decision Mamba (DMamba) integrates the Mamba framework, a selective structured state space model, into the Decision Transformer architecture for offline reinforcement learning. The model achieves competitive performance on D4RL continuous control tasks, even surpassing baselines on some datasets like Hopper-m-e with 111.1±0.3 expert-normalized returns, while showing mixed results on Atari discrete control tasks.

Key insights

• Mamba's selective state space model can serve as a competitive token-mixing module for sequential decision-making in reinforcement learning, offering an architectural alternative to Transformer's self-attention.
• The channel-mixing (feedforward) layer, typically found in Transformer blocks, was found not to provide a substantial performance benefit in DMamba for D4RL tasks, suggesting the Mamba block's token-mixing is largely sufficient.
• DMamba demonstrates the ability to leverage longer context lengths for performance improvement in some tasks (e.g., Breakout), aligning with Mamba's strengths in handling long sequences, but longer contexts can also degrade performance in others (e.g., Qbert).

Summary by alphaXiv.