Reproduce “Generative AI Is Not Ready for Clinical Use in Patient Education for Lower Back Pain Patients, Even With Retrieval-Augmented Generation” on OpenResearch

About this paper

Yi-Fei Zhao, Allyn Bove, David Thompson, James Hill, Yi Xu, Yufan Ren, Andrea Hassman, Leming Zhou and 1 more

This study evaluated generative AI, specifically Large Language Models augmented with Retrieval-Augmented Generation (RAG) and few-shot learning, for creating personalized patient education materials for low back pain (LBP) patients. It found that while RAG improved content accuracy, reduced redundancy, and enhanced readability to a 5th-grade level, the generated materials currently lack the clinical specificity and granularity required for direct clinical use.

Key insights

• RAG significantly improved the accuracy, reduced redundancy, and dramatically enhanced the readability of LLM-generated patient education materials, addressing some limitations of base LLMs.
• Few-shot learning was effective in guiding LLMs to generate advice on specific topics (e.g., lifting mechanics) but could inadvertently reduce the overall completeness of the generated content by overemphasizing these areas.
• Despite quantitative improvements, the generated materials still lacked the necessary clinical specificity, granularity, and ability to account for individual patient deviations, making them unsuitable for direct clinical application at present.

Summary by alphaXiv.