Reproduce “HGCLIP: Exploring Vision-Language Models with Graph Representations for Hierarchical Understanding” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Peng Xia, Xingtong Yu, Ming Hu, Lie Ju, Zhiyong Wang, Peibo Duan, Zongyuan Ge

HGCLIP introduces a framework that enhances Vision-Language Models for hierarchical image classification by integrating multi-modal prompt learning with graph representations of class hierarchies. It establishes new state-of-the-art performance across 11 diverse visual recognition benchmarks, demonstrating improved accuracy and robustness to distribution shifts.

Key insights

  • Deeply integrating structured hierarchical knowledge via graph neural networks into Vision-Language Models significantly enhances their ability for hierarchical image classification across multiple granularity levels.
  • Combining multi-modal prompt learning with graph representations allows for trainable adaptation of VLMs to specific datasets, leading to superior accuracy and more separable feature clusters.
  • The framework exhibits strong robustness to domain and subpopulation shifts, and can leverage approximate hierarchies generated by Large Language Models (LLMs) to still improve classification accuracy.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.