Reproduce “SUM: Saliency Unification through Mamba for Visual Attention Modeling” on OpenResearch

About this paper

Alireza Hosseini, Amirhossein Kazerouni, Saeed Akhavan, Michael Brudno, Babak Taati

A new model, SUM, unifies saliency prediction across natural scenes, e-commerce, and user interfaces by integrating the efficient Mamba architecture with a novel conditional mechanism, achieving state-of-the-art results on 27 out of 30 benchmark metrics while maintaining high computational efficiency.

Key insights

• State Space Models, particularly Mamba, offer a computationally efficient alternative to Transformers for dense prediction tasks like saliency modeling, achieving linear complexity while effectively capturing long-range dependencies.
• Dynamic conditional modulation using learnable tokens within a C-VSS block enables a single model to robustly adapt and unify saliency prediction across distinct visual domains (natural scenes, e-commerce, UIs) without requiring specialized architectures.
• A multi-component loss function, optimizing for distribution similarity, linear correlation, overlap, fixation point accuracy, and pixel-wise error, is crucial for achieving superior and well-rounded saliency prediction performance.

Summary by alphaXiv.