Reproduce “Dynamic Group Transformer: A General Vision Transformer Backbone with Dynamic Group Attention” on OpenResearch

About this paper

Kai Liu, Tianyi Wu, Cong Liu, Guodong Guo

A Dynamic Group Transformer (DGT) architecture employs a dynamic group attention mechanism that adaptively partitions queries and selects relevant keys based on content, surpassing the limitations of fixed window-based attention in Vision Transformers. This approach yields state-of-the-art performance across major computer vision tasks like image classification, segmentation, and detection with competitive computational efficiency.

Key insights

• Dynamically grouping queries and selectively attending to content-relevant keys, rather than relying on fixed spatial windows, substantially improves attention flexibility and computational efficiency.
• Combining content-aware dynamic attention with established convolutional inductive biases (CPE, IRFFN) within a hierarchical Transformer framework creates a more powerful and versatile vision backbone.
• The ability of DG-Attention to model long-range, semantically relevant dependencies across spatially distant features enhances the understanding of global context within images, as evidenced by visualizations.

Summary by alphaXiv.