Reproduce “RotoGrad: Gradient Homogenization in Multitask Learning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Adrián Javaloy, Isabel Valera

An algorithm named RotoGrad addresses negative transfer in multitask learning by simultaneously homogenizing gradient magnitudes and aligning gradient directions across tasks using task-specific rotation matrices and dynamic scaling. It demonstrated superior and more stable performance on benchmarks such as NYUv2, CIFAR10, and CelebA compared to other state-of-the-art multitask optimization methods.

Key insights

  • Negative transfer in MTL requires a unified approach that simultaneously addresses both gradient magnitude and direction conflicts for comprehensive improvement.
  • Trainable rotation matrices applied to the shared feature space can effectively align task gradients and facilitate positive transfer, even in unconstrained optimization settings.
  • Dynamically balancing gradient magnitudes based on the relative convergence rates of tasks helps prevent any single task from monopolizing the shared learning capacity.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.