Reproduce “Dynamic Token Pruning in Plain Vision Transformers for Semantic Segmentation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Quan Tang, Bowen Zhang, Jiajun Liu, Fagui Liu, Yifan Liu
A method for dynamically pruning tokens in Vision Transformers for semantic segmentation reduces computational costs by 20% to 35% across various architectures and benchmarks. This approach achieves comparable or slightly improved mean Intersection over Union by processing "easy" tokens early and retaining essential contextual information.
Key insights
- Computation can be dynamically adapted to the input image's difficulty, allowing for early finalization of "easy" tokens and deeper processing for "hard" ones, mirroring human-like segmentation.
- Existing auxiliary loss blocks within ViT architectures can be repurposed for inference-time token difficulty grading, avoiding the need for additional, heavy modules for pruning.
- Preserving representative semantic context via a top-k strategy is critical for successful token pruning in dense prediction tasks, preventing accuracy degradation that might occur from complete removal of "easy" categories.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.