Reproduce “ACI-BENCH: a Novel Ambient Clinical Intelligence Dataset for Benchmarking Automatic Visit Note Generation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Wen-wai Yim, Yujuan Fu, Asma Ben Abacha, Neal Snider, Thomas Lin, Meliha Yetisgen

ACI-BENCH is introduced as the largest publicly available corpus for benchmarking automatic clinical note generation from doctor-patient visit dialogue, comprising 207 role-played encounters. The paper establishes baselines using transformer and large language models, finding that a division-based generation approach improves performance and that current models exhibit unexpected robustness to ASR transcription errors.

Key insights

  • Decomposing the complex task of clinical note generation into division-based subtasks significantly improves performance for transformer models, particularly for the OBJECTIVE and ASSESSMENT_AND_PLAN sections.
  • Large Language Models (LLMs) like GPT-4 demonstrate strong capabilities in capturing clinically relevant facts, achieving high MEDCON scores even without task-specific fine-tuning.
  • The study surprisingly found that ASR transcription errors had no remarkable impact on the note generation performance of the best-performing models, suggesting a certain robustness of these models to transcription inaccuracies.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.