Reproduce “Annotation Error Detection: Analyzing the Past and Present for a More Coherent Future” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jan-Christoph Klie, Bonnie Webber, Iryna Gurevych
The authors compare 18 annotation error detection methods across nine English datasets covering text, token, and span labeling. They find that annotation inconsistencies are harder to detect than clear errors. On CoNLL-2003 span labeling, Retag achieves flagger F1 of 0.32, versus 0.25 for Projection Ensemble. Detection prioritizes labels for human review; the study covers instance labels, not omitted spans.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.