Reproduce “Conversation AI Dialog for Medicare powered by Finetuning and Retrieval Augmented Generation” on OpenResearch

About this paper

Atharva Mangeshkumar Agrawal, Rutika Pandurang Shinde, Vasanth Kumar Bhukya, Ashmita Chakraborty, Sagar Bharat Shah, Tanmay Shukla, Sree Pradeep Kumar Relangi, Nilesh Mutyam

Researchers from multiple institutions conducted a comparative analysis of fine-tuning and Retrieval-Augmented Generation (RAG) approaches for conversational AI in doctor-patient dialogues within mixed medical domains. The study found that fine-tuned GPT excelled in fluency and structural coherence, while Llama-2 with RAG achieved superior semantic accuracy and relevance, significantly outperforming an LSTM baseline.

Key insights

• Transformer-based models (GPT, Llama) with multi-head attention and Rotary Positional Embeddings (RoPE) significantly improved long-range dependency capture compared to LSTMs for complex medical dialogues.
• Fine-tuning (especially GPT) helps models internalize conversational patterns and language style for high fluency and structural coherence.
• Retrieval-Augmented Generation (RAG) effectively grounds LLM responses in facts, leading to superior semantic accuracy and relevance by dynamically integrating external, domain-specific knowledge.

Summary by alphaXiv.