Reproduce “Enhanced Knowledge Distillation for Detection Transformer via Teacher Prediction Refinement” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yitong Xing, Yuhao Cheng, Yanping Li, Yichao Yan
Teacher Prediction Refinement Distillation (TPRD) improves knowledge distillation for DETR-style object detectors by correcting unreliable teacher predictions before they supervise a smaller student. Across COCO and Pascal VOC, it consistently raises detection accuracy, adding up to 0.8 AP on COCO and 2.2 AP50 on VOC, with no inference-time parameters or latency and less than 20% training-time overhead.
Key insights
- Teacher-target quality is a distinct issue from teacher–student alignment: later decoder outputs can be less reliable than earlier outputs for the same query.
- Intermediate decoder predictions provide useful corrective signals, but replacing every logit discards useful class relationships; target-logit-only replacement performs better.
- TPRD is designed as a plug-in enhancement for multiple DETR distillation frameworks and adds no deployment cost, although its refinement uses ground-truth labels and extra training computation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.