Reproduce “IndicVisionBench: Benchmarking Cultural and Multilingual Understanding in VLMs” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Ali Faraz, Akash, Shaharukh Khan, Raja Kolla, Akshat Patidar, Suranjan Goswami, Abhinav Ravi, Chandra Khatri and 1 more

Krutrim AI researchers developed IndicVisionBench, the first large-scale benchmark designed to evaluate Vision-Language Models (VLMs) on their understanding of diverse Indian culture and multiple Indic languages. The benchmark, covering VQA, MMT, and OCR, revealed significant performance gaps in state-of-the-art VLMs in culturally nuanced and low-resource language settings.

Key insights

  • The benchmark's adversarial VQA questions, designed to incorporate false assumptions, proved particularly challenging for all evaluated models, indicating a need for more robust cultural reasoning.
  • Evaluation on the VQA-Parallel corpus demonstrated that visual input is crucial for effective VLM performance, as removing it led to substantial accuracy drops.
  • The consistent difficulty across models with certain Indic languages (e.g., Odia) and specific cultural topics highlights persistent biases and limitations in current VLM capabilities beyond high-resource contexts.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.