Reproduce “Crocodile: Cross Experts Covariance for Disentangled Learning in Multi-Domain Recommendation” on OpenResearch

About this paper

Zhutian Lin, Junwei Pan, Haibin Yu, Xi Xiao, Ximei Wang, Zhixiang Feng, Shifeng Wen, Shudong Huang and 2 more

The "Crocodile" model introduces a multi-domain recommendation system that resolves the conflict between capturing distinct domain-specific user interests and ensuring sufficient parameter learning for data-scarce domains. It achieves this by combining shared multi-embeddings with a Cross-experts Covariance Loss and a Prior Informed Element-wise Gating mechanism, leading to improved performance on public datasets and a 0.72% CTR lift and 0.73% GMV lift in online A/B tests on Tencent's advertising platform.

Key insights

• The fundamental dilemma in Multi-Domain Learning (MDL) involves balancing the need for domain-specific distinctiveness with the challenge of insufficient parameter learning in long-tailed data distributions.
• Explicitly minimizing the covariance between expert outputs (disentanglement) is an effective mechanism to ensure experts capture diverse and non-redundant aspects of user preferences across domains, even when using shared parameters.
• Using shared multi-embeddings, rather than strictly domain-specific ones, can prevent dimensional collapse in small domains while still allowing for domain-aware learning when combined with appropriate disentanglement and gating mechanisms.

Summary by alphaXiv.