Reproduce “Efficient Training of Language Models to Fill in the Middle” on OpenResearch

About this paper

Mohammad Bavarian, Heewoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey, Jerry Tworek, Mark Chen

The authors train causal language models to fill missing spans using right context. They split examples into prefix, middle, and suffix, then reorder some so the middle follows both surrounding parts during next-token training. With 50% transformed examples, left-to-right loss trends remain similar; on random-span HumanEval infilling, character-level training reaches 0.321 pass rate versus 0.015 for line-level training.

Key insights

• Autoregressive language models can acquire Fill in the Middle (FIM) capabilities during pretraining without degrading their standard left-to-right generation performance, a property termed "FIM-for-Free".
• Higher FIM training rates (up to 90-100%) and the Suffix-Prefix-Middle (SPM) data transformation structure generally yield better infilling performance.
• Integrating FIM during pretraining is computationally more efficient than finetuning existing models for FIM, as finetuning requires significant resources and can temporarily degrade standard autoregressive performance.

Summary by alphaXiv.