Reproduce “CM1 -- A Dataset for Evaluating Few-Shot Information Extraction with Large Vision Language Models” on OpenResearch

About this paper

Fabian Wolf, Oliver Tüselmann, Arthur Matei, Lukas Hennies, Christoph Rass, Gernot A. Fink

Researchers at TU Dortmund and Osnabrück University developed CM1, a dataset of historical administrative forms, to benchmark the few-shot information extraction performance of Large Vision Language Models (LVLMs) and specialized models on complex documents with handwritten entries. The work compares their efficacy in extracting key-value information, such as names and birthdates, under varying data availability.

Key insights

• LVLMs (PaliGemma, Qwen2.5-VL) consistently demonstrated superior generalization and significantly outperformed DONUT in few-shot settings (1% to 25% training data).
• Despite the few-shot strength of LVLMs, the smaller, specialized DONUT model achieved competitive or superior performance when fine-tuned with the full 100% training dataset.
• Zero-shot evaluation with large commercial LVLMs like ChatGPT-4o revealed inadequate performance for robust information extraction from historical handwriting without domain-specific fine-tuning.

Summary by alphaXiv.