Reproduce “DuwatBench: Bridging Language and Visual Heritage through an Arabic Calligraphy Benchmark for Multimodal Understanding” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Shubham Patle, Sara Ghaboura, Hania Tariq, Mohammad Usman Khan, Omkar Thawakar, Rao Muhammad Anwer, Salman Khan

DuwatBench provides a new benchmark for multimodal understanding of Arabic calligraphy, featuring 1,272 real-world images across six styles with comprehensive word/sentence-level transcriptions and bounding box annotations. Evaluations on leading multimodal models reveal that closed-source systems, particularly Gemini-2.5-Flash, show better performance (e.g., 71.82 chrF and 0.42 ExactMatch), but all models still struggle with the artistic complexities of calligraphic script, especially ornate styles.

Key insights

  • Multimodal models, including state-of-the-art LMMs, still face substantial challenges in accurately transcribing stylized Arabic calligraphy, with an average ExactMatch across models below 0.18.
  • Closed-source models (e.g., Gemini-2.5-Flash) consistently outperform open-source counterparts, demonstrating greater resilience to calligraphic irregularities and achieving higher accuracy (e.g., Gemini-2.5-Flash with 71.82 chrF and 0.42 ExactMatch).
  • Model performance varies significantly by calligraphic style; Naskh and Ruq’ah are easier, while ornate (Diwani, Thuluth) and geometric (Kufic) styles remain highly challenging, often leading to incoherent outputs or cultural biases.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.