Reproduce “Dynamic Convolution: Attention over Convolution Kernels” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yinpeng Chen, Xiyang Dai, Mengchen Liu, Dongdong Chen, Lu Yuan, Zicheng Liu
Microsoft researchers developed Dynamic Convolution, an operator that enhances the representation capability of lightweight convolutional neural networks by aggregating multiple kernels with input-dependent attention. This method achieves consistent accuracy improvements, such as a 5.6% increase in Top-1 ImageNet accuracy for MobileNetV2, with only a marginal 4% rise in computational cost.
Key insights
- Aggregating multiple convolution kernels with a "sum-to-one" attention constraint (via softmax) simplifies the learning process for attention models in deep dynamic networks.
- Using a large softmax temperature, annealed during early training epochs, promotes a more uniform attention distribution, allowing for more effective joint optimization of multiple kernels.
- Dynamic convolution effectively increases model complexity and representation power in lightweight networks without needing to increase their depth or width, addressing inherent underfitting.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.