Reproduce “Cross-Layer Retrospective Retrieving via Layer Attention” on OpenResearch

About this paper

Yanwen Fang, Yuxi Cai, Jintai Chen, Jingyu Zhao, Guangjian Tian, Guodong Li

Researchers from The University of Hong Kong, Zhejiang University, and Huawei Noah's Ark Lab developed Multi-head Recurrent Layer Attention (MRLA), a mechanism enabling neural network layers to selectively retrieve information from preceding layers based on relevance. This approach consistently enhanced performance across various CNNs and Vision Transformers on ImageNet classification (e.g., +1.6% Top-1 accuracy for ResNet-50) and COCO object detection (e.g., +4.0% box AP for Faster R-CNN), with a light-weighted variant achieving linear computational complexity for deep networks.

Key insights

• Explicitly applying attention across layers, treating layer outputs as "tokens" for retrieval, provides a new paradigm for enhancing information flow and representation learning in deep networks.
• A linear approximation within the recurrent layer attention framework (MRLA-light) effectively addresses the scalability challenge, making cross-layer attention practical for very deep architectures with O(T) complexity.
• The adaptive weighting of information from previous layers, guided by a learnable parameter (lambda), is crucial for optimizing the integration of multi-level hierarchical features.

Summary by alphaXiv.