Reproduce “Can Continual Pre-training Bridge the Performance Gap between General-purpose and Specialized Language Models in the Medical Domain?” on OpenResearch

About this paper

Niclas Doll, Jasper Schulze Buschhoff, Shalaka Satheesh, Hammam Abdelwahab, Héctor Allende-Cid, Katrin Klug

This research investigates whether continual pre-training and model merging can enable 7B language models to compete with larger general-purpose models on complex German medical tasks. It constructs a specialized German medical corpus (FineMed-de) and demonstrates that adapted 7B models can significantly narrow the performance gap, achieving a 3.5-fold increase in win-rate against a 24B model on instruction-following tasks.

Key insights

• A hybrid data filtering methodology leveraging both LLMs and traditional ML effectively creates high-quality, domain-specific corpora from general web data, addressing data scarcity in specialized non-English domains.
• Continual pre-training and model merging can significantly close the performance gap between smaller (7B) specialized LLMs and larger (24B) general-purpose models on complex medical instruction-following tasks.
• Domain adaptation introduces specific trade-offs; while it improves factual grounding and reduces certain failure modes (e.g., hallucination), it can also introduce new issues like increased language mixing or verbosity, suggesting limitations in current merging techniques.
• Optimal domain adaptation strategies may vary with model scale, as the applied method was beneficial for 7B models but detrimental for the 24B model in instruction-following capabilities.

Summary by alphaXiv.