Reproduce “Vision-Language Models Create Cross-Modal Task Representations” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Grace Luo, Trevor Darrell, Amir Bar
Researchers at UC Berkeley reveal that Vision-Language Models (VLMs) form shared, modality-invariant "task vectors" in their latent space, compressing task information from text examples, image examples, or instructions into a consistent representation. This work demonstrates that injecting these compressed task vectors, even from a different modality, significantly improves cross-modal task performance by 14-33% compared to standard few-shot prompting.
Key insights
- VLMs create compressed, high-level summaries of tasks, termed "task vectors," that are invariant to the input modality (text or image) and format (examples or instructions).
- Task representations are largely preserved from base Language Models (LLMs) during VLM fine-tuning, indicating efficient knowledge transfer and re-use of language functions.
- The internal processing of tasks in VLMs follows a consistent three-phase evolution (Input, Task, Answer) for both text and image in-context learning, where task summaries emerge in middle layers.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.