Reproduce “Not All Tokens Are Equal: Human-centric Visual Analysis via Token Clustering Transformer” on OpenResearch

About this paper

Wang Zeng, Sheng Jin, Wentao Liu, Chen Qian, Ping Luo, Wanli Ouyang, Xiaogang Wang

TCFormer introduces a novel Vision Transformer architecture that dynamically generates and merges vision tokens through semantic clustering, adapting their shapes and resolutions to visual importance. This approach achieves state-of-the-art performance across human-centric tasks like whole-body pose estimation, 3D human mesh reconstruction, and face keypoint localization, while remaining competitive in general image classification.

Key insights

• Dynamically generated tokens, adapting their shapes and resolutions based on semantic meaning and visual importance, significantly improve performance in fine-grained human-centric analysis tasks.
• Clustering tokens based on feature similarity allows for flexible, non-rectangular token shapes that better align with the irregular contours of human body parts.
• Aggregating features in the token domain via the MTA head is crucial for effectively preserving fine details across hierarchical stages, particularly for dense prediction tasks.
• The DPC-KNN clustering algorithm, as opposed to simpler importance-based selection, ensures a broader and more effective coverage of important regions for token merging.

Summary by alphaXiv.