Reproduce “Agglomerative Token Clustering” on OpenResearch

About this paper

Joakim Bruslund Haurum, Sergio Escalera, Graham W. Taylor, Thomas B. Moeslund

Agglomerative Token Clustering (ATC) introduces a parameter-free, hard-merging approach for Vision Transformers based on bottom-up hierarchical clustering, improving efficiency by reducing token count. The method consistently outperforms previous token reduction techniques across image classification, synthesis, and object detection/segmentation, demonstrating performance gains up to 10.5 percentage points in MAE-pretrained models and achieving better FID scores and mAP, particularly at aggressive token reduction rates.

Key insights

• Bottom-up hierarchical clustering creates more semantically coherent and informative token groupings by iteratively merging the most similar features first, preserving diverse features longer.
• Replacing bipartite matching in prior merging methods with agglomerative clustering overcomes architectural limitations, allowing for effective token reduction at aggressive keep rates below 50%.
• Average and complete linkage functions generally outperform single linkage in agglomerative clustering for token reduction, particularly at lower token keep rates.

Summary by alphaXiv.