Reproduce “From Handwriting to Structured Data: Benchmarking AI Digitisation of Handwritten Forms” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Nicholas Pather, Joshua Fouché, Sitwala Mundia, Karl-Günter Technau, Thokozile Malaba, Alex Welte, Ushma Mehta, Bruce A. Bassett
A benchmark of multimodal AI for converting handwritten South African maternity records into structured JSON finds that leading frontier models reach about 85% overall field accuracy on 49 real forms, but performance varies sharply by field type. Explicit schema constraints improve rare-category classification, while free-text transcription and hallucination remain too error-prone for unsupervised clinical data entry.
Key insights
- Field type strongly determines reliability: categorical selections are generally easier, formatted values show mixed model rankings, and free-text transcription is consistently the weakest task.
- Enumerating valid categorical values in the prompt improves macro F1 by roughly 0.21–0.29 for leading models, helping uncommon classes more than overall weighted scores.
- Model choice should be field-specific: Gemini 3.1 Pro led discrete and free-text extraction, GPT 5.4 had the lowest hallucination rate at 6%, and Claude Sonnet 4.6 was strongest on formatted fields.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.