Reproduce “Medicine on the Edge: Comparative Performance Analysis of On-Device LLMs for Clinical Reasoning” on OpenResearch

About this paper

Leon Nissen, Philipp Zagar, Vishnu Ravi, Aydin Zahedivash, Lara Marie Reimer, Stephan Jonas, Oliver Aalami, Paul Schmiedmayer

Researchers from Stanford University and University Hospital Bonn performed a comparative analysis of 13 on-device large language models (LLMs) for clinical reasoning across six Apple mobile devices, evaluating their accuracy, computational efficiency, and thermal limitations using the open-ended AMEGA benchmark. The study revealed that medically fine-tuned models like Aloe 8B and Med42 8B achieved top accuracy scores around 490 points, while the iPad Pro (M4) demonstrated throughputs up to 120 tokens per second.

Key insights

• Model size is not the sole determinant of performance; factors such as training data quality, diversity, and underlying base model architecture significantly influence accuracy and speed on MLX-based inference.
• Memory capacity is the primary bottleneck for on-device LLM deployment on mobile devices; iOS's strict memory limits often prevent smaller models from running despite theoretical parameter compatibility.
• Medically fine-tuned models like Aloe 8B and Med42 8B achieved the highest clinical reasoning accuracy, while the non-medical Phi 3 Mini 3.8B offered a strong balance of accuracy and efficiency, outperforming several larger or specialized models.

Summary by alphaXiv.