Reproduce “Sparse DETR: Efficient End-to-End Object Detection with Learnable Sparsity” on OpenResearch

About this paper

Byungseok Roh, JaeWoong Shin, Wuhyun Shin, Saehoon Kim

Researchers from KakaoBrain and Lunit developed Sparse DETR, an end-to-end object detector that employs learnable sparsity in its Transformer encoder to boost efficiency. This method achieved 48.2 AP with only 10% encoder tokens on the Swin-T backbone, reducing computational cost by 38% and increasing inference speed by 42% compared to Deformable DETR+.

Key insights

• Saliency determined by the Decoder Cross-Attention Map (DAM) provides a more effective signal for encoder token selection than objectness scores, leading to better performance at high sparsity.
• Auxiliary supervision within intermediate encoder layers is crucial for stable training and performance gains in deep Transformer encoders, effectively mitigating vanishing gradients.
• Substantial computational efficiency can be achieved in end-to-end detectors by intelligently pruning redundant encoder tokens without sacrificing, and often improving, detection accuracy.

Summary by alphaXiv.