Reproduce “Deep Incubation: Training Large Models by Divide-and-Conquering” on OpenResearch

About this paper

Zanlin Ni, Yulin Wang, Jiangwei Yu, Haojun Jiang, Yue Cao, Gao Huang

Deep Incubation introduces a modular training strategy for large deep learning models, employing a divide-and-conquer approach guided by a shallow meta-model to ensure compatibility among independently trained modules. This method improves generalization performance, reduces training costs by up to 4.1x for similar accuracy, and enhances data efficiency, also enabling the effective training of deeper and narrower architectures.

Key insights

• A shallow meta-model, pre-trained and then fixed, can effectively guide the compatibility of independently trained modules, resolving the conflict between efficiency and guidance effectiveness.
• Training individual modules by optimizing an end-to-end task loss within a hybrid network (Module Incubation) fosters better cooperation and performance than direct imitation of meta-modules.
• The proposed modular training approach effectively mitigates overfitting, leading to increased data efficiency and enabling the successful training of significantly deeper model architectures.

Summary by alphaXiv.