Reproduce “Instruction-Guided Fusion of Multi-Layer Visual Features in Large Vision-Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xu Li, Yi Zheng, Haotian Chen, Xiaolei Chen, Yuxuan Liang, Chenghang Lai, Bin Li, Xiangyang Xue
The method combines vision-encoder features from multiple layers, using the text instruction to weight layer groups before sending fused features to the language model. This changes information carried by each visual token without increasing token count, though the aggregator adds computation. On LLaVA-v1.5-7B, it scores 1519.8 on MME-p versus 1480.6 for baseline and leads on eight of ten benchmarks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.