Reproduce “FlexPipe: Adapting Dynamic LLM Serving Through Inflight Pipeline Refactoring in Fragmented Serverless Clusters” on OpenResearch

About this paper

Yanying Lin, Shijie Peng, Chengzhi Lu, Chengzhong Xu, Kejiang Ye

FlexPipe adapts large-language-model serving pipelines while inference continues, splitting stages more finely for bursts and merging them during stable traffic. It profiles operators, monitors queues and variability, and transfers KV-cache state during topology changes. On a cluster evaluation at CV=1, it reduced end-to-end latency 38.3% versus AlpaServe and ServerlessLLM with identical 12,000-request goodput; this result is workload-specific.

Key insights

• Resource fragmentation in cloud environments significantly impedes communication-intensive parallelism strategies, necessitating alternative approaches for distributed LLM inference.
• Pipeline granularity presents a fundamental trade-off: fine-grained offers elasticity for bursty workloads, while coarse-grained minimizes communication overhead for stable periods, requiring dynamic transitions.
• Request distribution variability causes pipeline stage imbalances, and deeper pipeline architectures can absorb peak loads through distributed buffering in highly bursty environments.

Summary by alphaXiv.