Reproduce “In-context Learning Distillation: Transferring Few-shot Learning Ability of Pre-trained Language Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yukun Huang, Yanda Chen, Zhou Yu, Kathleen McKeown

Researchers at Columbia University developed In-context Learning Distillation (ICL-D) to transfer the few-shot learning capabilities of large language models to smaller models. Their method enables students to perform multitask few-shot learning, with one distilled model being 13.4x smaller and 3.6x faster while retaining 91.4% of the teacher's performance.

Key insights

  • Multitask-ICT, a direct tuning approach on target few-shot examples, consistently outperforms Meta-ICT and traditional supervised fine-tuning for few-shot learning.
  • Distillation of in-context learning consistently improves student models' few-shot performance across various settings.
  • General linguistic knowledge transferred via language modeling distillation is crucial and complementary to in-context learning objectives for Multitask-ICT, but less so for Meta-ICT.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.