Reproduce “CLEME2.0: Towards Interpretable Evaluation by Disentangling Edits for Grammatical Error Correction” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jingheng Ye, Zishan Xu, Yinghui Li, Linlin Song, Qingyu Zhou, Hai-Tao Zheng, Ying Shen, Wenhao Jiang and 4 more
CLEME2.0, developed by researchers at Tsinghua University and collaborators, is an evaluation metric for Grammatical Error Correction that disentangles edits into specific scores for hit-correction, wrong-correction, under-correction, and over-correction. This metric achieves superior correlation with human judgments and demonstrates enhanced robustness across diverse datasets by integrating semantic-aware edit weighting.
Key insights
- Disentangling edits into specific categories provides actionable, diagnostic feedback, enabling GEC system developers to precisely identify and address weaknesses in hit-correction, wrong-correction, under-correction, or over-correction.
- Incorporating semantic importance through edit weighting significantly improves the alignment between automatic GEC evaluation metrics and nuanced human judgments.
- A robust GEC evaluation metric must maintain high correlation with human judgments consistently across diverse reference datasets with varying annotation styles to ensure generalizability.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.