Reproduce “Evaluation of HTR models without Ground Truth Material” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Phillip Benjamin Ströbel, Simon Clematide, Martin Volk, Raphael Schwitter, Tobias Hodel, David Schoch

Researchers at the Universities of Zurich and Bern developed and evaluated metrics for assessing Handwritten Text Recognition (HTR) output quality without requiring ground truth data. Their findings demonstrate that masked language models, particularly multilingual BERT, and high-order character n-gram ratios reliably predict HTR accuracy and effectively aid in selecting optimal HTR models for historical documents.

Key insights

  • Masked Language Models (MLMs), specifically pre-trained multilingual BERT, are highly effective and accessible for estimating HTR quality without requiring domain-specific language model training.
  • High-order character n-gram ratios (e.g., 7-gram) and token ratios provide robust and reliable indicators of HTR quality, performing comparably to advanced MLMs.
  • The evaluated ground truth-free metrics can accurately identify and rank HTR models, offering a practical solution for model selection in real-world digitization projects.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.