Reproduce “From Generalist to Specialist: Improving Large Language Models for Medical Physics Using ARCoT” on OpenResearch

About this paper

Jace Grandinetti, Rafe McBeth

Researchers from the University of Pennsylvania developed ARCoT, an Adaptable Retrieval-based Chain of Thought framework, to enhance Large Language Model (LLM) accuracy in medical physics without costly fine-tuning. The framework boosts GPT-4's performance from 67% to 90% on a benchmark exam, exceeding the average human score of 68%.

Key insights

• The ARCoT framework achieves significant accuracy improvements (47% on average over base LLMs) in specialized domains without the need for model retraining or fine-tuning, making it cost-effective and accessible.
• Combining RAG with advanced prompting techniques like Step-Back and Chain-of-Thought creates a synergistic effect, notably improving LLM reasoning and factuality beyond RAG alone.
• ARCoT successfully transforms generalist LLMs into domain specialists, with GPT-4 achieving a 90% score on a medical physics exam, surpassing the average human performance of 68%.

Summary by alphaXiv.