Reproduce “LLMStructBench: Benchmarking Large Language Model Structured Data Extraction” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Sönke Tenckhoff, Mario Koddenbrock, Erik Rodner

Researchers from the University of Applied Sciences Berlin developed LLMStructBench, a benchmark comprising 995 manually verified samples, to evaluate Large Language Models' performance in structured data extraction and JSON generation. The study revealed that prompting strategies can be as influential as model size in ensuring parsing reliability, with open-source models like Gemma3-27B achieving competitive performance against proprietary systems.

Key insights

  • The choice of prompting strategy can be as influential as, or more influential than, model size in ensuring parsing reliability and structural validity, especially for smaller models.
  • A trade-off exists where schema enforcement via certain prompting strategies (e.g., PJ+) can guarantee parseable outputs but often leads to an increased number of semantic errors (Wrong Values).
  • While model size generally correlates with improved value accuracy (`F1_micro`), document-level correctness (`DOC_micro`) is more sensitive to the specific compatibility between the chosen prompt and the model architecture.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.