Reproduce “ELA: Efficient Local Attention for Deep Convolutional Neural Networks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Wei Xu, Yi Wan
The Efficient Local Attention (ELA) module enhances deep convolutional neural networks by efficiently encoding spatial information through lightweight operations and Group Normalization, improving model focus without reducing channel dimensionality. It consistently improves performance across image classification, object detection, and semantic segmentation tasks, demonstrating gains such as a 2.39% increase in MobileNetV2 top-1 accuracy on ImageNet.
Key insights
- Generating attention by independently processing horizontal and vertical feature vectors allows for effective spatial information encoding.
- Using 1D convolutions for attention generation is more computationally efficient and better suited for sequential signals than 2D convolutions.
- Replacing Batch Normalization with Group Normalization enhances the module's robustness and generalization, especially in diverse training conditions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.