Reproduce “MedRealMM: A Real-World Multimodal Benchmark for Chinese Online Medical Consultation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Runhan Shi, Quan Zhou, Yuqian Xu, Shuai Yang, Xin Wu, Zitong Zhou, Hui Liu, Bin Zha and 9 more

MedRealMM introduces a benchmark of 5,620 real-world, de-identified multimodal Chinese online medical consultation cases, featuring patient-uploaded images and multi-turn dialogues. It employs a physician-guided, case-specific rubric to evaluate open-ended LLM responses, revealing that even frontier models achieve only around 50% of human physician performance, largely due to failures in safety-critical error avoidance and integrating visual evidence.

Key insights

  • Human physicians' superior performance in real-world consultations is primarily driven by their ability to avoid unsafe, unsupported, or contradictory recommendations, rather than solely by maximizing positive clinical actions.
  • Patient-uploaded images are critical for effective clinical reasoning, with multimodal inputs leading to a substantial 14-20 point performance increase for frontier models, highlighting a major capability gap in text-only medical AI.
  • Current LLMs exhibit significant weaknesses in 'history taking' competence (the 'know-when-to-ask' skill) and struggle particularly in specialized domains like Traditional Chinese Medicine and psychiatry.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.