Reproduce “Distributed Inference and Fine-tuning of Large Language Models Over The Internet” on OpenResearch

About this paper

Alexander Borzunov, Max Ryabinin, Artem Chumachenko, Dmitry Baranchuk, Tim Dettmers, Younes Belkada, Pavel Samygin, Colin Raffel

PETALS, a collaborative effort from institutions including HSE University and Hugging Face, enables users to run and fine-tune large language models like BLOOM (176B) and Llama 2 (70B) by distributing computation across multiple devices connected over the internet. This approach significantly reduces per-device hardware requirements, achieving up to a 10x performance improvement over local offloading for interactive inference and maintaining stability through novel fault-tolerance and load-balancing mechanisms.

Key insights

• Large Language Model (LLM) computation can be effectively distributed over unreliable internet connections, allowing individual participants to host only a small portion of a massive model.
• Robust fault tolerance and dynamic load balancing are crucial for maintaining performance and availability in geographically dispersed, unreliably connected distributed systems.
• By enabling parameter-efficient fine-tuning on distributed models, users can customize LLMs without needing to host the entire model or manage complex multi-GPU setups.

Summary by alphaXiv.