Reproduce “JDocQA: Japanese Document Question Answering Dataset for Generative Language Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Eri Onami, Shuhei Kurita, Taiki Miyanishi, Taro Watanabe

Researchers from Nara Institute of Science and Technology and RIKEN introduced JDocQA, a large-scale Japanese document question answering dataset designed for generative and multimodal language models. This dataset mandates the integration of visual and textual information, including unanswerable questions to assess hallucination, and demonstrated that finetuned models outperform general baselines, with visual inputs enhancing multimodal performance.

Key insights

  • Finetuning generative language models on domain-specific, multimodal datasets like JDocQA substantially improves their performance on document question answering tasks compared to zero-shot general models.
  • Incorporating unanswerable questions into the training data helps models learn to identify when information is missing and correctly decline to answer, thereby contributing to the mitigation of hallucination.
  • Multimodal models benefit significantly from the inclusion of specific visual inputs (e.g., bounding box cropped images), confirming that integrated visual and textual understanding is crucial for complex document comprehension.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.