Reproduce “MedNLI Is Not Immune: Natural Language Inference Artifacts in the Clinical Domain” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Christine Herlihy, Rachel Rudinger
Researchers at the University of Maryland confirmed the presence of annotation artifacts in MedNLI, a clinical Natural Language Inference dataset, identifying specific lexical and semantic patterns that enable models to achieve inflated performance on superficial cues. Adversarial filtering demonstrated that models relying on these artifacts experienced significant performance drops on examples requiring deeper inference.
Key insights
- Annotation artifacts are a pervasive issue, found even in specialized datasets like MedNLI, which are expertly annotated by domain specialists.
- Expert annotators, such as physicians, inadvertently introduce systematic "design patterns" or heuristics (e.g., hypernyms for entailment, explicit negation for contradiction) during hypothesis generation.
- Models trained on such datasets learn to exploit these shallow cues, leading to inflated performance metrics and a lack of genuine inferential capability on truly challenging instances.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.