Reproduce “KIEval: Evaluation Metric for Document Key Information Extraction” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Minsoo Khang, Sang Chul Jung, Sungrae Park, Teakgyu Hong
Upstage AI researchers developed KIEval, a new evaluation metric for Document Key Information Extraction that accounts for structured information and aligns error quantification with industrial correction costs. KIEval demonstrates that conventional F1 overestimates performance in tasks requiring structural understanding and provides a more accurate assessment for real-world applications.
Key insights
- Conventional Entity F1 scores consistently overstate KIE model performance on grouped datasets by giving credit for individual entity extraction even when structural relationships are incorrect.
- The total 'correction cost' required in industrial scenarios is often significantly different from and frequently lower than the traditional sum of False Positives and False Negatives, highlighting a misalignment of existing metrics.
- KIEval reveals nuanced differences in 'structure awareness' among multimodal Large Language Models, showing that some LLMs extract entities well but struggle with correct grouping.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.