Reproduce “Stress Testing BERT Anaphora Resolution Models for Reaction Extraction in Chemical Patents” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Chieling Yueh, Evangelos Kanoulas, Bruno Martins, Camilo Thorne, Saber Akhondi
A study by researchers at the University of Amsterdam and Elsevier evaluated the robustness of BERT-based anaphora resolution models for reaction extraction in chemical patents. The work found that OCR-induced typographical errors significantly reduced model performance (up to 6.9% Macro F1 drop for 50% Word Error Rate), while named entity recognition (NER) span boundary errors had a comparatively minor impact, and incorporating simulated OCR noise into training data improved robustness on noisy test sets by up to 3.8% Macro F1.
Key insights
- OCR-induced typographical errors lead to a substantial performance degradation in anaphora resolution models, decreasing Macro F1 by up to 6.9% at 50% WER.
- Anaphora resolution models are comparatively robust to NER span boundary mistakes, showing only minor performance drops (max 0.8% Macro F1).
- Training models with simulated OCR errors can improve robustness against such noise by up to 3.8% Macro F1 on noisy datasets, although it may slightly reduce performance on clean data.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.