Reproduce “Introducing Cluster Efficiency to LLM Serving: Pooling Economical Accelerators over Fast Interconnects” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Youhe Jiang, Pavle Glušac, Binhang Yuan, Eiko Yoneki
The authors evaluate Galaxy, a cluster of lower-end accelerators, for long-context language-model serving. The system pools chips over fast links and uses communication-aware tensor parallelism and decode optimizations to reduce coordination costs. Across 36 model, precision, and context configurations, Galaxy exceeds an eight-GPU server in tokens per dollar and usually tokens per joule; TCP-class link emulation erases all dollar-efficiency wins.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.