Reproduce “Making Pre-trained Language Models Better Few-shot Learners” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Tianyu Gao, Adam Fisch, Danqi Chen

This paper presents LM-BFF, a framework that enhances few-shot learning capabilities of moderately-sized pre-trained language models by integrating prompt-based fine-tuning with automatic prompt generation and refined in-context demonstration strategies. The approach achieves up to 30% absolute improvement on tasks like SNLI and an 11% average absolute improvement across 15 NLP tasks compared to standard fine-tuning, using minimal annotated data.

Key insights

  • Prompt-based fine-tuning, by reformulating tasks as MLM problems, significantly boosts few-shot performance for moderately-sized LMs.
  • Automatic prompt generation can effectively identify or create templates and label words that match or surpass the performance of manually designed prompts.
  • In-context demonstrations are most effective when strategically sampled, with semantically similar examples providing superior guidance compared to random selections.
  • Parameter updates (fine-tuning) are essential for smaller LMs to leverage few-shot demonstrations, unlike very large models which can rely on in-context learning alone.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.