Reproduce “From Vision to Language: Investigating Causal Information Flow in Multimodal Decision-Making” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Davide Testa, Hugh Mee Wong, Alessandro Lenci, Bernardo Magnini, Albert Gatt
Researchers introduced a causal intervention framework, Attention Knockout, to analyze how Vision-Language Models (VLMs) integrate visual information for video-based decision-making. The study found that visual grounding primarily occurs during the processing of candidate answer options and highlighted that temporal reasoning poses greater challenges for VLMs due to fragile sequential information processing.
Key insights
- Visual information is primarily integrated and utilized when VLMs process the textual content of candidate answer options, functioning as the main site for cross-modal grounding in late processing layers.
- Nouns consistently act as semantic anchors for grounding visually identifiable entities across different reasoning tasks, while verbs show increased dependence on visual input for temporal reasoning due to event dynamics.
- Temporal reasoning presents more fragile processing dynamics for VLMs, struggling with sequential event order and being significantly influenced by linguistic biases (e.g., preference for 'Before' over 'After'), unlike spatial and causal reasoning.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.