Reproduce “Common Objects Out of Context (COOCo): Investigating Multimodal Context and Semantic Scene Violations in Referential Communication” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Filippo Merlo, Ece Takmaz, Wenkai Chen, Albert Gatt
An investigation into how Vision-Language Models (VLMs) process visual context and semantic violations during object naming reveals an adaptive strategy where models leverage congruent context but are hindered by incongruent context. Researchers introduced the COOCo dataset, systematically manipulating object-scene semantic relatedness and visual noise, and demonstrated VLM attention patterns align with human cognitive observations.
Key insights
- VLMs demonstrate an adaptive strategy: they leverage congruent scene context to aid object identification but treat semantically incongruent context as a distraction.
- The models' attention allocation to target objects exhibits a non-monotonic, U-shaped dependency on semantic fit, with both highly expected and highly incongruent objects attracting increased attention, mirroring human cognitive patterns.
- Mid-layers (7-15) of the vision encoder are critical for target-focused semantic processing and cross-modal fusion, with attention allocation to the target increasing in these layers for correct predictions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.