Reproduce “Scheduling Data to Compute: Hierarchical Data Pipelining for LLM Training on AMD GPUs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Gengyu Rao, Ghadeer Ahmed H Alabandi, Su Ann Chong, Thomas Huber, Kailash Gogineni, Subhajit Dutta Chowdhury, Saleel Kudchadker, Suchita Pati and 23 more
The authors present an AMD system coordinating data movement and computation for LLM training. Within GPUs, tensor data movers stage and transpose data; across GPUs, DMA engines transfer it without assigning transfers to compute units. Tested compressed sparse attention kernels show a 1.88× training-step speedup, while overlap tests show reduced compute slowdowns, not universal end-to-end training gains.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.