Reproduce “Towards Resource-Efficient Serverless LLM Inference with SLINFER” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Chuhao Xu, Zijun Li, Quan Chen, Han Zhao, Xueyan Tang, Minyi Guo
SLINFER, a system developed by researchers at Shanghai Jiao Tong University and Nanyang Technological University, optimizes serverless inference for small- to mid-sized Large Language Models by transparently leveraging heterogeneous CPU and GPU resources. It achieves 47% to 154% more SLO-met requests compared to baselines by enabling elastic, on-demand resource sharing.
Key insights
- Modern CPUs with matrix accelerators can independently serve small- to mid-sized LLMs while meeting production SLOs, offering a valuable, underutilized resource in heterogeneous data centers.
- Co-locating multiple LLM instances on a single CPU or GPU can significantly improve resource utilization due to the often-undersaturated nature of individual LLM workloads in serverless environments.
- Fine-grained, token-level scheduling with predictive "shadow validation" and hazard-aware memory management are crucial for balancing dynamic LLM resource demands with strict SLOs and avoiding memory failures.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.