Reproduce “LiSA: Linear Stochastic Attention for Long-Range Visual Modeling” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Sahar Dastani, Fereshteh Shakeri, Samuel Barbeau, Kunal Mahatha, Ismail Ben Ayed, Herve Lombaert, Christian Desrosiers
LiSA adds truncated multi-hop information flow to linear attention for vision, while a local branch preserves neighborhood detail. It computes higher-order walk terms without forming a full token-by-token attention matrix, retaining linear scaling in token count for fixed truncation length. On ImageNet-1K at 224×224, LiSA-PVT-T reaches 78.53% Top-1 accuracy, above FLatten-PVT-T’s 77.8%, with modestly higher reported FLOPs.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.