Reproduce “Large Language Model-Based Generation of Discharge Summaries” on OpenResearch

About this paper

Tiago Rodrigues, Carla Teixeira Lopes

Researchers from the University of Porto evaluated five large language models (LLMs) for automatically generating discharge summaries from extensive clinical notes, demonstrating that proprietary models like Gemini 1.5 Pro and GPT-4, especially with one-shot prompting, produce higher quality summaries quantitatively and qualitatively compared to open-source alternatives. The study identified persistent challenges in abstractive summarization, including factual inconsistencies and the difficulty of inferring information not explicitly present in source notes.

Key insights

• One-shot prompting significantly improves the performance of proprietary LLMs (GPT-4, Gemini 1.5 Pro) in generating clinical summaries, leading to higher similarity and quality scores.
• Fine-tuning open-source models (Mistral-7B, Llama2-7B) with QLoRA yielded minimal performance gains for this specific task, highlighting the challenges with smaller models and context constraints.
• Despite advancements, LLMs still face challenges with factual hallucinations, achieving low exact-match scores, and inferring information (e.g., discharge instructions) that is typically determined post-admission.

Summary by alphaXiv.