Reproduce “EdgeLoRA: An Efficient Multi-Tenant LLM Serving System on Edge Devices” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zheyu Shen, Yexiao He, Ziyao Wang, Yuning Zhang, Guoheng Sun, Wanghao Ye, Ang Li
EdgeLoRA, developed by the University of Maryland, College Park, is an efficient multi-tenant LLM serving system for edge devices specifically designed for Low-Rank Adaptation (LoRA) adapters. It achieves a 2-4x throughput boost and can serve orders of magnitude more LoRA adapters compared to existing frameworks like `llama.cpp`.
Key insights
- Dynamically selecting the optimal LoRA adapter via an adaptive router can lead to better overall performance than fixed, manually chosen adapters.
- Efficient memory management using a combination of LRU caching and pre-allocated pools is crucial for scaling multi-LoRA serving on memory-constrained edge devices.
- Batching inference requests with different LoRA adapters significantly improves GPU utilization and reduces latency in dynamic multi-tenant edge environments.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.