Reproduce “FactEHR: A Dataset for Evaluating Factuality in Clinical Notes Using LLMs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Monica Munnangi, Akshay Swaminathan, Jason Alan Fries, Jenelle Jindal, Sanjana Narayanan, Ivan Lopez, Lucia Tu, Philip Chung and 3 more
Researchers from Stanford University developed FactEHR, a large-scale dataset and evaluation framework for assessing the factual accuracy of Large Language Models in diverse clinical notes. The study found considerable variability in LLM performance for fact decomposition, with leading models like GPT-4o demonstrating higher precision but still struggling with comprehensive recall from dense clinical documentation.
Key insights
- LLM performance for fact decomposition and verification varies substantially across models and clinical note types; leading closed models consistently outperform smaller open-source models in accuracy and consistency.
- GPT-4o effectively serves as a robust proxy for human judgment in clinical textual entailment tasks, exhibiting high recall (0.96) and precision (0.86) against expert annotations, thus enabling scalable factuality evaluation.
- While LLMs generally achieve high fact precision (accuracy of generated facts), their fact recall (completeness of extracted information) remains a challenge, indicating difficulty in capturing all relevant details from dense clinical texts.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.