Reproduce “Revisiting Token Pruning for Object Detection and Instance Segmentation” on OpenResearch

About this paper

Yifei Liu, Mathias Gehrig, Nico Messikommer, Marco Cannici, Davide Scaramuzza

Researchers from the University of Zurich's Robotics and Perception Group developed Selective Vision Transformer (SViT), an efficient token pruning method for Vision Transformers performing object detection and instance segmentation. SViT achieves up to 34% faster inference for the entire network on COCO 2017 with only a 0.3 mAP degradation, by preserving and reactivating pruned tokens with a dynamic pruning rate.

Key insights

• For dense prediction tasks, preserving the features of pruned tokens within feature maps is crucial, as discarding them leads to substantial performance degradation.
• The ability to reactivate previously pruned tokens in later layers significantly improves robustness and allows the model to adapt to the dynamic relevance of information across different processing depths.
• A simple 2-layer Multi-Layer Perceptron (MLP) is sufficient for effective token selection, reducing the overhead typically associated with more complex gating networks.
• Employing a dynamic pruning rate that adapts to the input image's content yields better performance compared to a fixed pruning rate across all inputs.

Summary by alphaXiv.