Reproduce “CLoCKDistill: Consistent Location-and-Context-aware Knowledge Distillation for DETRs” on OpenResearch

About this paper

Qizhen Lan, Qing Tian

CLoCKDistill, a knowledge distillation framework, enhances the performance of smaller student Detection Transformer (DETR) models by leveraging global contextual information from the teacher's encoder memory and utilizing ground truth-guided queries for consistent logit distillation. The approach yielded up to a 6.4% mean Average Precision improvement for DINO ResNet-18 student models on the KITTI dataset and a 2.9% improvement on COCO, outperforming prior state-of-the-art distillation methods.

Key insights

• Distilling the global contextual 'memory' from the Transformer encoder is more effective for DETRs than distilling local features from the CNN backbone, as it better captures long-range dependencies.
• Guiding distillation with ground truth location and scale information (via masks and target-aware queries) ensures the student focuses on relevant object features and balances contributions from different object sizes and foreground/background ratios.
• Creating 'target-aware' queries directly from ground truth provides stable and precise distillation points for logit transfer, overcoming the instability associated with unordered object queries in DETR architectures.

Summary by alphaXiv.