Reproduce “FuseFlow: A Fusion-Centric Compilation Framework for Sparse Deep Learning on Streaming Dataflow” on OpenResearch

About this paper

Rubens Lacouture, Nathan Zhang, Ritvik Sharma, Marco Siracusa, Fredrik Kjolstad, Kunle Olukotun, Olivia Hsu

FuseFlow is a compilation framework designed for sparse deep learning models on streaming dataflow architectures. It introduces comprehensive cross-expression kernel fusion and enhanced intra-expression iteration fusion, demonstrating performance speedups ranging from 1.5x to 3.9x over unfused baselines across various sparse deep learning models.

Key insights

• General cross-expression fusion is crucial for optimizing sparse deep learning on dataflow architectures, significantly reducing intermediate memory transfers and improving efficiency.
• The optimal fusion granularity for sparse models is application-dependent; partial fusion often outperforms full fusion by balancing memory reduction with recomputation overhead.
• Factored iteration spaces, which interleave input iteration and computation, are more efficient for sparse machine learning on dataflow architectures compared to fully-fused global iteration.

Summary by alphaXiv.