Reproduce “Lessons from Natural Language Inference in the Clinical Domain” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Alexey Romanov, Chaitanya Shivade

Researchers from UMass Lowell and IBM Almaden developed MedNLI, the first publicly available, expert-annotated dataset for Natural Language Inference in the clinical domain. They demonstrated that neural network models, particularly when utilizing domain-specific word embeddings and sequential transfer learning, achieve up to 73.5% accuracy on this challenging task, with a novel knowledge-directed attention mechanism showing modest gains.

Key insights

  • Domain-specific word embeddings, particularly those trained on clinical notes, are crucial for adapting NLI models to the clinical domain, yielding substantial performance improvements over general-domain embeddings.
  • Sequential transfer learning effectively leverages large open-domain NLI datasets to enhance performance on limited clinical data, whereas direct transfer of pre-trained models is largely ineffective.
  • Simple retrofitting of word embeddings using complex ontological relationships can unexpectedly degrade performance, indicating that nuanced, multi-step reasoning often required for clinical inference goes beyond direct semantic proximity.
  • A knowledge-directed attention mechanism provides a more effective way to integrate multi-step ontological relationships, offering slight performance gains in complex clinical reasoning tasks.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.