Reproduce “Disaggregated Multi-Tower: Topology-aware Modeling Technique for Efficient Large-Scale Recommendation” on OpenResearch

About this paper

Liang Luo, Buyun Zhang, Michael Tsang, Yinbin Ma, Ching-Hsiang Chu, Yuxin Chen, Shen Li, Yuchen Hao and 6 more

Researchers at Meta AI introduced Disaggregated Multi-Tower (DMT), a topology-aware modeling technique for large-scale recommendation systems that optimizes training by aligning model architecture with data center network hierarchies. The method achieved up to 1.9x training throughput speedup for DLRM and 1.8x for DCN on various hardware configurations, while maintaining or improving model quality.

Key insights

• The fundamental inefficiency in large-scale recommendation model training stems from a mismatch between model architectures, global training paradigms, and the hierarchical communication topology of data centers.
• Global communication operations can be effectively decomposed and re-orchestrated into smaller, topology-aware collectives and local exchanges to reduce network bottlenecks without compromising model semantics.
• Introducing hierarchical feature interactions via Tower Modules can reduce both cross-host communication volume and computational complexity, potentially enhancing model effectiveness through more efficient interaction patterns.

Summary by alphaXiv.