Reproduce “MangaVQA and MangaLMM: A Benchmark and Specialized Model for Multimodal Manga Understanding” on OpenResearch

About this paper

Jeonghun Baek, Kazuki Egashira, Shota Onohara, Atsuyuki Miyai, Yuki Imajuku, Hikaru Ikuta, Kiyoharu Aizawa

Researchers from The University of Tokyo introduce MangaVQA and MangaOCR, two new benchmarks for evaluating large multimodal models' (LMMs) ability to understand manga by focusing on page-level text recognition and contextual visual question answering. They also developed MangaLMM, a specialized LMM that achieves competitive performance on these benchmarks, outperforming many general-purpose models.

Key insights

• General-purpose LMMs, including state-of-the-art proprietary models, demonstrate near-zero explicit OCR performance on manga but can implicitly capture some textual semantics to aid VQA.
• Joint fine-tuning on both OCR and VQA tasks for MangaLMM effectively handles both, with enhanced OCR capabilities providing a slight synergistic benefit to VQA performance.
• High-quality synthetic VQA training data, generated by LLMs guided by explicit OCR annotations, is critical for improving LMM performance on complex VQA tasks in specialized domains.

Summary by alphaXiv.