Reproduce “Vidur: A Large-Scale Simulation Framework For LLM Inference” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Amey Agrawal, Nitin Kedia, Jayashree Mohan, Ashish Panwar, Nipun Kwatra, Bhargav Gulavani, Ramachandran Ramjee, Alexey Tumanov
Vidur helps providers choose LLM serving configurations by profiling selected operations, predicting unprofiled runtimes, and simulating request scheduling on CPUs. Vidur-Search uses these simulations to select setups maximizing queries per second per dollar under latency constraints. For LLaMA2-70B, transferring a configuration optimal for Arxiv-Summarization-4K to LMSys-Chat-1M incurs 2× cost overhead; prediction accuracy is workload- and capacity-dependent.
Key insights
- Most LLMs share common architectural properties, allowing for decomposition into a small set of generic operators whose runtimes can be efficiently profiled and estimated.
- Optimal LLM inference deployment configurations are highly dependent on the specific model and the characteristics of the application workload, such as prompt and decode lengths.
- Operation triaging, classifying operators by their runtime dependencies (token-level, sequence-level, communication), enables tailored and accurate runtime prediction from limited profiling data.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.