Reproduce “LUMEN: Coordinated Failure Recovery for Distributed LLM Serving” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhang Cao, Shujie Han, Juncheng Zhang, Yuanming Ren, Yongkun Li, Patrick P. C. Lee
LUMEN is a distributed LLM serving system that orchestrates failure recovery by using load-aware checkpoint placement, interrupted-request dispatch, and progressive recovery. It demonstrated significant reductions in both Time-To-First-Token (TTFT) and Time-Per-Output-Token (TPOT) latencies and overall recovery time across various failure rates and cluster sizes, without introducing overhead during normal operations.
Key insights
- Efficient LLM failure recovery can be framed as a load-aware coordination problem, requiring dynamic decisions for checkpoint placement, interrupted-request distribution, and serving capacity restoration.
- Proactively distributing KV cache checkpoints and dynamically rebalancing recovery load are crucial for mitigating performance degradation and avoiding bottlenecks during failures.
- Utilizing recovering workers with a draft model and speculative decoding during their full model reload phase significantly reduces the impact of lost serving capacity on cluster-wide latency.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.