Reproduce “Exploring deep learning for Event-Based Saliency Prediction with a Transformer-based model” on OpenResearch

About this paper

Romaric Mazna, Jean Martinet, Sai Deepesh Pokala

Researchers from i3S/CNRS, Université Côte d’Azur developed SEST, a transformer-based model for event-based saliency prediction, which is the first deep learning approach in this domain. They also introduced new large-scale synthetic datasets, N-DHF1K and N-UCF Sports, to address data scarcity, and demonstrated SEST's improved performance over existing event-based methods as well as its zero-shot transferability to real event data.

Key insights

• Event data, when represented as volumetric voxel grids and processed by transformer architectures, allows for effective dynamic saliency prediction, establishing a deep learning baseline.
• Synthetic event-based datasets derived from existing RGB saliency benchmarks can serve as viable training data to bootstrap deep learning models for event cameras, addressing data scarcity.
• The explicit modeling of spatio-temporal information via 3D convolutions in the decoder and the inclusion of a learnable center-bias component are important for improving saliency map accuracy in event-based models.

Summary by alphaXiv.