Reproduce “Efficient Masked Attention Transformer for Few-Shot Classification and Segmentation” on OpenResearch

About this paper

Dustin Carrión-Ojeda, Stefan Roth, Simone Schaub-Meyer

The Efficient Masked Attention Transformer (EMAT) introduces a memory-efficient masked attention mechanism and learnable downscaling for few-shot classification and segmentation. It improves segmentation accuracy, especially for small objects, achieving up to +2.3% mIoU gain on the smallest objects compared to the state-of-the-art, while using approximately four times fewer trainable parameters.

Key insights

• Reformulating masked attention to operate only on unmasked entries can drastically reduce memory consumption, enabling the use of higher-resolution features critical for small object segmentation.
• A hybrid learnable downscaling approach provides more effective feature compression than traditional pooling, preserving finer details needed for accurate segmentation at higher resolutions.
• Reduced model parameters, combined with memory-efficient attention and learnable downscaling, enhance generalization capabilities and improve performance on challenging tasks like small object segmentation within few-shot learning.

Summary by alphaXiv.