Reproduce “Deep Multi-task Representation Learning: A Tensor Factorisation Approach” on OpenResearch

About this paper

Yongxin Yang, Timothy Hospedales

Researchers at Queen Mary, University of London developed Deep Multi-Task Representation Learning (DMTRL), a framework that uses tensor factorization to automatically learn layer-wise knowledge sharing within deep neural networks for multi-task learning. The framework consistently outperforms single-task learning and often surpasses user-defined deep MTL, particularly with limited data and across heterogeneous tasks, achieving lower error rates (e.g., below 6% on MNIST 1-vs-all).

Key insights

• Tensor factorization provides a data-driven mechanism to automatically discover layer-wise knowledge sharing structures within deep networks, removing the need for manual, heuristic architectural design.
• The approach enables a continuous spectrum of knowledge sharing, allowing for more nuanced and adaptive knowledge transfer between tasks compared to traditional binary (shared/not shared) decisions.
• Learned sharing patterns often reveal higher sharing strength in lower (feature extraction) layers and progressively less sharing in higher (task-specific) layers, which aligns with expected functional specialization but is discovered automatically.

Summary by alphaXiv.