Reproduce “Content-aware Token Sharing for Efficient Semantic Segmentation with Vision Transformers” on OpenResearch

About this paper

Chenyang Lu, Daan de Geus, Gijs Dubbelman

Researchers at Eindhoven University of Technology introduce Content-aware Token Sharing (CTS), a framework that boosts the computational efficiency of Vision Transformer-based semantic segmentation by intelligently sharing tokens among semantically uniform image patches. This approach enabled up to a 105% throughput increase on the Cityscapes dataset and a nearly 64% increase on state-of-the-art BEiT-L + UPerNet models while preserving segmentation quality.

Key insights

• Exploiting semantic redundancy by sharing tokens among uniform neighboring image patches can substantially improve the computational efficiency of ViT-based semantic segmentation without degrading performance.
• An efficient, class-agnostic policy network is more effective for identifying shareable tokens than relying on full semantic segmentation predictions, offering a balance between accuracy and computational overhead for policy generation.
• The simple, reversible token sharing and unsharing mechanism allows the framework to be architecture-agnostic, broadly compatible with diverse ViT backbones, pre-training strategies, and segmentation decoders.

Summary by alphaXiv.