Reproduce “Towards Conversational Medical AI with Eyes, Ears and a Voice” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Meet Shah, Jason Gusdorf, Anil Palepu, Chunjong Park, Jack W. O'Sullivan, Vishnu Ravi, Tim Strother, Pavel Dubov and 45 more

The authors present an audiovisual telehealth AI co-clinician that pairs a patient-facing Talker with a supervisory Clinical Planner to guide live conversation and track clinical goals. In 120 simulated encounters across 20 outpatient scenarios, its case-specific triage score was 91.67%, matching PCPs, while physical-exam and red-flag scores were lower. These simulated results do not establish clinical equivalence or safety.

Key insights

  • A dual-agent architecture, separating conversational mechanics (Talker) from supervisory clinical reasoning (Clinical Planner), is crucial for balancing real-time responsiveness with systematic medical rigor in multimodal AI systems.
  • AI evaluation in healthcare requires novel frameworks beyond traditional rubrics, specifically to identify unique AI failure modes like "contextual completion" (inferring findings without direct observation) and expose safety-critical omissions.
  • Current AI capabilities present a "jagged technological frontier," demonstrating strong performance in some clinical reasoning aspects while consistently lagging in nuanced areas such as comprehensive physical examinations and proactive red-flag detection.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.