Reproduce “GradTS: A Gradient-Based Automatic Auxiliary Task Selection Method Based on Transformer Networks” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Weicheng Ma, Renze Lou, Kai Zhang, Lili Wang, Soroush Vosoughi

GradTS uses task-specific Transformer attention-head gradients to identify auxiliary NLP tasks and training examples that are likely to benefit a primary task. Across GLUE and additional mixed-task settings, its trial-based and instance-filtering variants generally outperform selection baselines such as AUTOSEM while requiring less search time, with especially large gains on small or difficult datasets such as RTE, WNLI, MELD, and Dyadic-MELD.

Key insights

  • Overlap in which Transformer attention heads matter most provides a practical proxy for task relatedness that can reveal useful connections missed by broad task labels or dataset statistics.
  • Combining gradient-based pre-ranking with empirical validation preserves flexibility in the final task set while substantially reducing the search space compared with exhaustive selection.
  • Filtering individual auxiliary examples is valuable because task relatedness does not imply that every example in the task supports the primary objective.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.