Reproduce “V-SEAM: Visual Semantic Editing and Attention Modulating for Causal Interpretability of Vision-Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Qidong Wang, Junjie Hu, Ming Jiang
V-SEAM tests whether vision-language models rely on particular image facts by editing a question-relevant region and measuring changes in attention-head outputs. Activation patching identifies positive- and negative-effect heads for amplification or suppression during inference, without changing model parameters. On binary GQA questions, rescaling raises LLaVA average accuracy from 84.72% to 89.48%; results are limited to these tested tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.