Reproduce “Generalist Foundation Models Are Not Clinical Enough for Hospital Operations” on OpenResearch

About this paper

Lavender Y. Jiang, Angelica Chen, Xu Han, Xujin Chris Liu, Radhika Dua, Kevin Eaton, Frederick Wolff, Robert Steele and 11 more

Researchers at NYU Langone Health developed Lang1, a family of specialized language models for hospital operations, demonstrating that these models, pretrained on in-domain electronic health records and finetuned, outperform much larger generalist foundation models by 3.64% to 6.75% AUROC on five critical tasks. The study also introduced ReMedE, a new benchmark designed to evaluate model utility for real-world clinical operational decisions, revealing that generalist models often perform poorly in zero-shot settings on these tasks.

Key insights

• Generalist foundation models, even those with hundreds of billions of parameters, exhibit poor zero-shot performance on hospital operational tasks, indicating a fundamental gap between general medical knowledge and operational utility.
• While in-domain pretraining does not lead to emergent predictive capabilities for operational tasks in a zero-shot manner, it significantly enhances the efficiency and performance of subsequent supervised finetuning.
• Smaller, specialized models pretrained on proprietary clinical data and then finetuned can dramatically outperform much larger generalist models for specific, high-stakes operational tasks, suggesting a 'specialist' AI approach is superior for these applications.

Summary by alphaXiv.