Reproduce “Rethinking Hard-Parameter Sharing in Multi-Domain Learning” on OpenResearch

About this paper

Lijun Zhang, Qizheng Yang, Xiao Liu, Hui Guan

An empirical study by the University of Massachusetts Amherst re-evaluates hard parameter sharing in multi-domain learning, finding that making bottom layers domain-specific consistently outperforms the conventional strategy of sharing bottom layers. Models employing this "bottom-specific" approach achieve competitive or superior performance to independent models with significantly fewer domain-specific parameters.

Key insights

• The conventional wisdom of sharing bottom layers and specializing top layers for multi-domain learning is not optimal and is consistently outperformed by specializing bottom layers and sharing top layers.
• Bottom layers benefit significantly from domain-specific parameters, suggesting that even low-level feature extraction needs to be tailored for diverse visual domains.
• Placing domain-specific capacity in the bottom layers leads to a more efficient and higher-performing hard parameter sharing strategy for multi-domain learning.

Summary by alphaXiv.