Reproduce “Hierarchical Pre-Training of Vision Encoders with Large Language Model” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Eugene Lee, Ting-Yu Chang, Jui-Huang Tsai, Jiajie Diao, Chen-Yi Lee

HIVE introduces hierarchical cross-attention during vision-language pretraining, connecting selected intermediate layers of a vision encoder to corresponding LLM layers instead of exposing only final image features. The approach yields modest classification gains, improves multimodal benchmarks—most notably OK-VQA, from 46.19 to 51.01 over the self-attention baseline—and reduces reported training time and peak memory relative to full self-attention.

Key insights

  • Intermediate visual representations can provide complementary information: earlier layers preserve local structure, while deeper layers capture broader semantic content.
  • Hierarchical cross-attention is most clearly beneficial for multimodal question answering and reasoning, with smaller and inconsistent effects on image classification.
  • Qualitative gradient and attention visualizations suggest that the method delivers more localized supervision to early vision layers and broader semantic attention at later depths.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.