Reproduce “Distilling Large Language Models for Efficient Clinical Information Extraction” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Karthik S. Vedula, Annika Gupta, Akshay Swaminathan, Ivan Lopez, Suhana Bedi, Nigam H. Shah
Researchers at Stanford University and collaborators successfully applied knowledge distillation to develop efficient BERT-based models for clinical information extraction, achieving comparable performance to large language models (LLMs) while dramatically reducing computational cost and inference time. These distilled models process clinical notes 12 times faster and are up to 101 times cheaper than LLM teachers, enabling practical deployment in healthcare.
Key insights
- Knowledge distillation effectively transfers high-performance capabilities from large, computationally expensive LLMs to smaller, efficient BERT models, achieving competitive accuracy for clinical NER with substantial resource savings.
- The most effective ensemble of teacher labelers (LLMs and ontologies) varies by specific NER task, with ontology inclusion sometimes negatively impacting performance due to increased false positives.
- Error analysis revealed significant inconsistencies and inaccuracies in existing human-labeled gold standard datasets, suggesting that the reported performance of the distilled models may be an underestimate.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.