Reproduce “Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li and 1 more

T5 treats language tasks as text inputs paired with generated outputs, using task prefixes and fine-tuning a pre-trained model for each downstream task. It pre-trains with span corruption, replacing corrupted spans with sentinels and generating their missing text. T5-11B reaches 88.9 on SuperGLUE’s aggregate, versus the cited prior best of 84.6; this result reflects a bundled recipe, not an isolated component.

Key insights

  • The encoder-decoder Transformer architecture generally performed best for text-to-text tasks, and sharing parameters between the encoder and decoder offered efficiency without performance loss.
  • Most denoising objectives performed similarly, with sentinel-token span corruption being slightly more efficient and effective, and a 15% corruption rate providing a good balance.
  • Heuristic cleaning of large web corpora (like C4) was essential for performance, and fine-tuning all model parameters consistently outperformed parameter-efficient methods like adapter layers.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.