Reproduce “Instruction Tuning with GPT-4” on OpenResearch

About this paper

Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, Jianfeng Gao

Microsoft Research developed LLaMA-GPT4 by fine-tuning LLaMA models with 52,000 English and Chinese instruction-following examples generated by GPT-4, demonstrating that a state-of-the-art teacher model can significantly enhance the zero-shot instruction-following capabilities of open-source LLMs. The resulting LLaMA-GPT4 model achieved performance comparable to the original GPT-4 on human evaluation criteria for helpfulness, honesty, and harmlessness on unseen tasks, and outperformed prior instruction-tuned open-source models.

Key insights

• A more powerful teacher model (GPT-4) produces significantly higher quality instruction-following data, leading to superior alignment and performance in open-source LLMs compared to data from less capable teachers (e.g., GPT-3.5).
• Instruction tuning with GPT-4-generated data can bring the performance of a smaller, open-source model (LLaMA 7B) remarkably close to that of the teacher model (GPT-4) on zero-shot instruction-following tasks.
• GPT-4 can effectively act as an automatic evaluator and generate high-quality comparison data, enabling scalable and cost-effective research into Reinforcement Learning from Machine Feedback (RLMF) for aligning open-source models.

Summary by alphaXiv.