Reproduce “Hierarchy-Aware Fine-Tuning of Vision-Language Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Jiayu Li, Rajesh Gangireddy, Samet Akcay, Wei Cheng, Juhua Hu

A framework for Vision-Language Models is presented that enhances hierarchical classification performance by integrating Low-Rank Adaptation with Tree-Path KL Divergence and Hierarchy-Sibling Smoothed Cross-Entropy losses. This approach yields significantly improved accuracy and consistency in multi-level categorization tasks while maintaining computational efficiency.

Key insights

  • The two custom loss functions, Tree-Path KL Divergence and Hierarchy-Sibling Smoothed Cross-Entropy, are complementary and exhibit a synergistic effect, jointly optimizing for both vertical and horizontal hierarchical consistency.
  • Integrating these hierarchy-aware losses directly within the VLM's shared embedding space effectively reshapes the space, causing fine-grained labels to cluster meaningfully around their coarse-level ancestors.
  • Parameter-efficient fine-tuning with LoRA maintains the VLM's powerful generalization capabilities while making the adaptation for complex hierarchical tasks computationally feasible.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.