Reproduce “Towards Resiliency in Large Language Model Serving with KevlarFlow” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shangshu Qian, Kipling Liu, P. C. Sruthi, Lin Tan, Yongle Zhang
KevlarFlow introduces a fault-tolerant architecture for Large Language Model serving, transitioning from a "fail-stop" to a "fail-stutter" model to enable continuous request processing during node failures. The system significantly reduces mean-time-to-recovery by 20x to around 30 seconds and improves p99 time-to-first-token by up to 574.6x while maintaining low operational overhead.
Key insights
- Decoupling the collective communication communicator's configuration from model weight loading enables dynamic runtime reconfiguration of serving topology for fault recovery and scaling.
- Treating multi-node pipeline parallelism as independent fault domains allows for containing failures locally and preserving partial serving capacity through dynamic traffic rerouting.
- Proactive, low-overhead background replication of the KV cache transforms catastrophic failures into seamless request migrations, preserving user session context and reducing latency spikes.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.