Reproduce “Multi-Task Dense Prediction via Mixture of Low-Rank Experts” on OpenResearch

About this paper

Yuqi Yang, Peng-Tao Jiang, Qibin Hou, Hao Zhang, Jinwei Chen, Bo Li

This paper presents MLoRE, a multi-task dense prediction framework that enhances Mixture-of-Experts (MoE) by explicitly modeling global task relationships and employing efficient low-rank experts. The framework achieves state-of-the-art performance on PASCAL-Context and NYUD-v2 datasets while improving parameter and computational efficiency through re-parameterization at inference.

Key insights

• Explicitly modeling global relationships among all tasks via a generic shared path is crucial for multi-task dense prediction, as standard Mixture-of-Experts (MoE) often fails to capture these comprehensive inter-task connections.
• Utilizing low-rank experts with varied ranks effectively enhances model capacity and feature diversity while significantly reducing parameters and computational cost, outperforming standard MoE approaches.
• Linearity within the expert paths enables re-parameterization at inference time, collapsing the complex multi-expert structure into a single, efficient convolutional layer, thereby improving deployment efficiency.

Summary by alphaXiv.