Reproduce “Enhancing Large Vision-Language Models with Layout Modality for Table Question Answering on Japanese Annual Securities Reports” on OpenResearch

About this paper

Hayato Aida, Kosuke Takahashi, Takahiro Omi

Researchers at Stockmark enhanced Large Vision-Language Models for table question answering on Japanese annual securities reports by incorporating in-table text and layout modality alongside visual input. This approach achieved an Accuracy of 0.9509 and an ANLS score of 0.9666 on a reformulated TableCellQA dataset, demonstrating superior performance over zero-shot LVLMs and nearing that of explicitly structured HTML input.

Key insights

• In-table textual content and layout information are crucial for accurate table understanding in LVLMs, with layout providing essential spatial context to correctly interpret table structures and avoid cell misidentification.
• While image modality can offer complementary character-level information, it can also introduce noise (e.g., OCR errors) that may slightly reduce exact match accuracy compared to text and layout alone.
• The multimodal approach effectively captures underlying table structure, achieving performance closely approximating explicitly structured HTML, making it a viable solution when structured data is unavailable.

Summary by alphaXiv.