Reproduce “FlexLLM: Token-Level Co-Serving of LLM Inference and Finetuning with SLO Guarantees” on OpenResearch

About this paper

Gabriele Oliaro, Xupeng Miao, Xinhao Cheng, Vineeth Kada, Mengdi Wu, Ruohan Gao, Yingyi Huang, Remi Delacourt and 4 more

FlexLLM introduces a system for co-serving LLM inference and PEFT-based finetuning on shared GPUs at the token level. This approach achieves a 1.9–6.8x increase in finetuning throughput and an 85–87% reduction in activation memory, all while preserving inference latency Service Level Objectives.

Key insights

• Fusing LLM inference and PEFT finetuning at the token level is critical for efficiently sharing GPU resources, as it allows fine-grained interleaving of operations while maintaining strict inference latency SLOs.
• Significant memory savings for finetuning activations can be achieved by pruning the backward graph to only compute gradients for PEFT parameters and selectively recomputing/compressing activations during backpropagation.
• Dynamic scheduling that adapts finetuning compute based on real-time inference load, utilizing idle GPU cycles opportunistically, effectively balances high finetuning throughput with guaranteed inference latency targets.

Summary by alphaXiv.