Reproduce “MLoRA: Multi-Domain Low-Rank Adaptive Network for CTR Prediction” on OpenResearch

About this paper

Zhiming Yang, Haining Gao, Dehong Gao, Luwei Yang, Libin Yang, Xiaoyan Cai, Wei Ning, Guannan Zhang

Researchers from Alibaba.COM introduce MLoRA (Multi-Domain Low-Rank Adaptive Network), which adapts the Low-Rank Adaptation (LoRA) technique for multi-domain Click-Through Rate prediction. This method efficiently learns domain-specific knowledge while maintaining a shared backbone, enhancing prediction performance across diverse domains with minimal additional parameters.

Key insights

• Model adaptations for diverse domains can be efficiently captured through low-rank decompositions, significantly reducing trainable parameters compared to full fine-tuning.
• Combining a shared backbone network with parameter-efficient, domain-specific low-rank adaptation modules effectively balances learning general patterns and capturing domain-specific nuances.
• A two-phase training strategy, involving pre-training the backbone on all data and then fine-tuning only the LoRA modules for each domain, optimizes the learning process.

Summary by alphaXiv.