Reproduce “ATTEMPT: Parameter-Efficient Multi-task Tuning via Attentional Mixtures of Soft Prompts” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Akari Asai, Mohammadreza Salehi, Matthew E. Peters, Hannaneh Hajishirzi

Researchers from the University of Washington and AI2 developed ATTEMPT, a parameter-efficient multi-task tuning method that leverages attentional mixtures of soft prompts to enhance language model performance. It achieves results comparable to or better than full fine-tuning while updating as little as 0.4% of the base model's parameters, demonstrating particular effectiveness in few-shot learning scenarios.

Key insights

  • Dynamically combining pre-trained source prompts with a target-specific prompt via an attention mechanism enables robust knowledge transfer, significantly boosting prompt tuning's performance and stability.
  • The learned attention mechanism provides interpretability by indicating which source tasks contribute most to a given target task, offering insights into task relationships.
  • A modular approach to multi-task learning allows flexible composition of knowledge from various source tasks, helping to mitigate issues like negative transfer.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.