Reproduce “Independent Component Alignment for Multi-Task Learning” on OpenResearch

About this paper

Dmitry Senushkin, Nikolay Patakin, Arseny Kuznetsov, Anton Konushin

Samsung Research developed a method called Aligned-MTL that addresses multi-task learning optimization challenges by introducing the condition number of the gradient matrix as a stability criterion. This approach, which uses Singular Value Decomposition for gradient alignment, consistently outperforms state-of-the-art methods across computer vision and reinforcement learning benchmarks while offering controlled convergence to a pre-defined task balance.

Key insights

• The condition number of the gradient matrix serves as a comprehensive metric for MTL optimization stability; a low condition number (ideally 1) implies equal gradient norms and orthogonality, addressing both dominance and conflicts.
• The gradient alignment problem can be framed as an Orthogonal Procrustes problem, solvable in closed-form via SVD, which enables principled modification of gradients.
• Aligned-MTL offers theoretical guarantees of linear convergence to a Pareto-stationary point that explicitly respects pre-defined task-specific weights, providing predictable control over the final task balance.

Summary by alphaXiv.