Reproduce “ElasticMM: Efficient Multimodal LLMs Serving with Elastic Multimodal Parallelism” on OpenResearch

About this paper

Zedong Liu, Shenggan Cheng, Guangming Tan, Yang You, Dingwen Tao

ElasticMM, a new serving system, enhances Multimodal Large Language Model (MLLM) inference efficiency by implementing Elastic Multimodal Parallelism, which dynamically manages resources across decoupled request types and inference stages. It achieves up to 4.2x lower Time-to-First-Token and 4.5x higher throughput compared to existing systems.

Key insights

• Hierarchical decoupling of MLLM serving by request modality and inference stage effectively mitigates resource contention and allows for tailored resource management.
• Dynamic and elastic resource allocation, adapted to both long-term workload patterns and sudden bursts, is critical for maintaining high efficiency and meeting Service-Level Objectives (SLOs) under heterogeneous MLLM traffic.
• Targeted multimodal optimizations, like a unified prefix cache and non-blocking encoding, are necessary to overcome unique MLLM overheads and improve latency beyond general LLM serving techniques.

Summary by alphaXiv.