Reproduce “UPDP: A Unified Progressive Depth Pruner for CNN and Vision Transformer” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ji Liu, Dehua Tang, Yuanxian Huang, Li Zhang, Xiaocheng Zeng, Dong Li, Mingjie Lu, Jinzhang Peng and 4 more
UPDP, a unified progressive depth pruning framework, optimizes both efficient CNNs and Vision Transformers by using a novel block modification strategy and a two-stage progressive training approach. The method achieves competitive accuracy with enhanced inference speed, demonstrating up to 2.50x speedup for MobileNetV2-1.4 and 4.9x for ConvNeXtV1-Tiny on ImageNet-1K.
Key insights
- Transforming diverse normalization layers (LayerNorm, GroupNorm) to BatchNorm, along with strategic activation layer insertions, allows for effective reparameterization and merging in depth pruning.
- A progressive training approach that gradually shifts from the baseline to the pruned network structure is crucial for maintaining high accuracy by smoothly transferring learned weights.
- Supernet training combined with a genetic algorithm for subnet searching efficiently explores the pruning space, leading to optimal pruned architectures that balance accuracy and speed.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.