Reproduce “BLEnD-Vis: Benchmarking Multimodal Cultural Understanding in Vision Language Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Bryan Chen Zhengyu Tan, Zheng Weihua, Zhengyuan Liu, Nancy F. Chen, Hwaran Lee, Kenny Tsu Wei Choo, Roy Ka-Wei Lee

The BLEnD-Vis framework introduces a multimodal, multicultural benchmark to assess the robustness and groundedness of everyday cultural knowledge in Vision-Language Models. It reveals that VLMs often struggle with linguistic rephrasing and exhibit low cross-modal consistency in understanding cultural facts, with performance disparities across regions correlating with digital representation.

Key insights

  • VLM performance on cultural understanding does not consistently correlate with model size; instead, architectural choices and training data diversity appear more influential.
  • Models exhibit brittle knowledge representations, showing performance degradation when cultural queries are linguistically rephrased, indicating reliance on specific phrasing rather than deep conceptual understanding.
  • Visual cues consistently improve VLM performance on cultural tasks, but cross-modal consistency (simultaneous correctness across text and vision) remains low, suggesting inadequate integration of textual and visual cultural knowledge.
  • Significant regional disparities persist in model performance, with lower-resource regions consistently showing reduced accuracy, aligning with biases in training data representation.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.