Reproduce “Sessa: Selective State Space Attention” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Liubomyr Horbatko
Researchers developed Sessa, a novel sequence modeling architecture that addresses limitations of Transformers and state-space models in long-context scenarios by integrating input-dependent attention within a recurrent feedback path. Sessa theoretically achieves slower, power-law memory decay and enables flexible selective retrieval profiles, including non-decaying or increasing influence from distant tokens, outperforming baselines on synthetic long-context tasks.
Key insights
- Sessa's feedback solve is Bounded-Input Bounded-Output (BIBO) stable under a uniform row contraction condition on the feedback matrix.
- Under diffuse routing conditions, Sessa exhibits a power-law memory decay (O(ℓ^-β_tail)) for long lags, which is asymptotically slower than the O(1/ℓ) decay in diffuse Transformers and exponential decay in 'failed freeze-time' SSMs.
- Deep Sessa networks can realize flexible selective retrieval profiles, including decaying, frozen (non-decaying, O(1)), and even increasing (O(ℓ^ν)) influence, a capability not matched by diffuse Transformers or 'failed freeze-time' Mamba models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.