Reproduce “CaseReportBench: An LLM Benchmark Dataset for Dense Information Extraction in Clinical Case Reports” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Xiao Yu Cindy Zhang, Carlos R. Ferreira, Francis Rossignol, Raymond T. Ng, Wyeth Wasserman, Jian Zhu

Researchers introduce CaseReportBench, an expert-annotated benchmark dataset for dense information extraction from clinical case reports, specifically focusing on rare diseases. The study evaluates Large Language Models (LLMs) on this benchmark, finding that models like Qwen2.5:7B with filtered category-specific prompting demonstrate high accuracy and efficiency, often outperforming larger proprietary models.

Key insights

  • Smaller, open-source LLMs like Qwen2.5:7B can achieve superior performance over larger proprietary models such as GPT-4o for dense medical information extraction, highlighting the importance of instruction fidelity and architectural design over sheer model size.
  • Filtered Category-Specific Prompting (FCSP), which segments reports and applies tailored prompts, offers comparable accuracy to Uniform Category-Specific Prompting (UCP) but with a 6.2% improvement in computational efficiency.
  • Few-Shot prompting consistently provides better performance compared to Zero-Shot or Zero-Shot Chain-of-Thought strategies for this dense information retrieval task.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.