Reproduce “[CLS] is Not Enough: Multi-Label Recognition via Patch-Level Inference and Adaptive Aggregation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Akang Wang, Xili Deng, Zhanxuan Hu, Yi Zhao, Yonghang Tai, Huafeng Li

The PIAA framework enhances multi-label image recognition by shifting from global image representations to fine-grained patch-level inference, then adaptively aggregating these localized predictions. It achieved state-of-the-art performance among training-free methods across four benchmarks, surpassing TagCLIP by 11.9% mAP on NUS-WIDE and delivering over 360x speedup in classifier acquisition compared to optimization-based unsupervised methods.

Key insights

  • Analytically derived unsupervised visual classifiers (PVCL) can effectively bridge the vision-language modality gap at the patch level, improving localized prediction reliability without relying on textual prototypes directly for patch classification.
  • Adaptive aggregation (PAA) through max-pooling and fusion with the global [CLS] token significantly enhances robustness by distilling discriminative signals and leveraging complementary global context for diverse object scales.
  • Shifting the VLM paradigm from a global [CLS] bottleneck to a patch-level inference and aggregation strategy can yield superior multi-label recognition performance and efficiency without requiring task-specific training.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.