Reproduce “DeAR: Fine-Grained VLM Adaptation by Decomposing Attention Head Roles” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yiming Ma, Hongkun Yang, Lionel Z. Wang, Bin Chen, Weizhi Xian, Jianzhi Teng
DeAR is a VLM adaptation framework that decomposes attention head roles in deeper Transformer layers, using a Concept Entropy metric to identify specialized "Attribute" and "Generalization" heads. Its Role-Based Attention Mask guides new task knowledge to relevant heads while preserving foundational capabilities, leading to a harmonic mean of 82.72% across 11 classification datasets and a 1.83% improvement in novel class performance over prior methods.
Key insights
- Functional specialization within Vision-Language Models, particularly in ViT-B-16, occurs at the fine-grained level of individual attention heads in deeper layers, not just between entire layers.
- Quantifying attention head roles using "Concept Entropy" allows for a data-driven classification of heads into specialized (e.g., attribute-focused) and generalist roles.
- Controlling information flow based on these identified attention head roles, via a "Role-Based Attention Mask," enables effective task adaptation while preserving the VLM's original zero-shot generalization capabilities.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.