Reproduce “FailSafe: High-performance Resilient Serving” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ziyi Xu, Zhiqiang Xie, Swapnil Gandhi, Christos Kozyrakis
FAILSAFE is a system designed for high-performance, resilient serving of large language models (LLMs) using tensor parallelism, specifically addressing GPU failures. It achieves up to 1.71x higher average throughput compared to baselines and reduces recovery latency from GPU failures by up to 183x, from 22 seconds to 120ms.
Key insights
- Persistent performance degradation after a GPU failure is largely due to compute and memory imbalances under irregular tensor parallelism configurations, which can be mitigated by intelligent resource management.
- Proactive KVCache backup to host memory combined with optimized, on-demand weight recovery drastically reduces recovery latency and eliminates expensive recomputation, improving user experience.
- A hybrid attention approach, combining tensor and data parallelism, effectively balances computation across irregularly configured GPUs by allowing replicated attention heads to process different requests.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.