Reproduce “MeTA-LoRA: Data-Efficient Multi-Task Fine-Tuning for Large Language Models” on OpenResearch

About this paper

Bo Cheng, Xu Wang, Jinda Liu, Yi Chang, Yuan Wu

META-LORA introduces a two-stage, meta-learning inspired framework for data-efficient multi-task LoRA fine-tuning of large language models. The method enables LLaMA2-13B to achieve a 46.32 BBH score using only 100 examples per task on Flanv2, outperforming multi-task LoRA baselines while using substantially less data and significantly reducing training time.

Key insights

• Integrating meta-learning principles with LoRA enables LLMs to 'learn how to learn' across tasks, drastically reducing the amount of task-specific data required for effective adaptation.
• The two-stage process effectively balances the capture of task-specific knowledge (Phase I) with the aggregation of generalizable meta-knowledge (Phase II), mitigating task interference issues.
• This approach allows for robust multi-task and multilingual performance with minimal data, making LLM fine-tuning more accessible for resource-constrained environments and niche applications.

Summary by alphaXiv.