Reproduce “Scalable Engine and the Performance of Different LLM Models in a SLURM based HPC architecture” on OpenResearch

About this paper

Anderson de Lima Luiz, Shubham Vijay Kurlekar, Munir Georges

Researchers at AImotion Bavaria and Technische Hochschule Ingolstadt developed and evaluated a SLURM-based High-Performance Computing (HPC) architecture for deploying heterogeneous Large Language Models (LLMs). This framework enables efficient resource management and scalable inference, handling various LLM sizes and supporting advanced workflows like RAG-based chatbots.

Key insights

• Smaller LLMs can sustain higher concurrency with lower latency, while larger models saturate with significantly fewer concurrent users and incur higher latencies.
• The architecture demonstrates minimal overhead from container spin-up and scheduling activities, confirming efficient resource dedication to inference tasks.
• Throughput generally decreases as concurrency increases due to the additional computational effort required for orchestrating and delivering requests across multiple GPUs and hardware resources.

Summary by alphaXiv.