Reproduce “SplitLLM: Collaborative Inference of LLMs for Model Placement and Throughput Optimization” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Akrit Mudvari, Yuang Jiang, Leandros Tassiulas
SplitLLM provides an optimal dynamic programming algorithm for distributing Large Language Model inference computations between a client and server in edge-cloud environments, minimizing server workload while adhering to latency agreements. This approach yielded an average 28.9% reduction in server-side computational load for Transformer models and drastically lowered server queue wait times in simulations.
Key insights
- LLM layers exhibit highly non-uniform computational demands, with self-attention layers showing quadratic complexity with sequence length, making granular, intelligent placement crucial.
- Optimal layer-wise splitting significantly reduces server workload without compromising model accuracy and is complementary to other model optimization techniques like sparse attention.
- Reframing the collaborative inference challenge as a constrained shortest path problem allows for a pseudo-polynomial time dynamic programming solution that guarantees optimal resource allocation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.