Reproduce “Hogwild! Inference: Parallel LLM Generation via Concurrent Attention” on OpenResearch

About this paper

Gleb Rodionov, Roman Garipov, Alina Shutova, George Yakushev, Erik Schultheis, Vage Egiazarian, Anton Sinitsin, Denis Kuznedelev and 1 more

Researchers from Yandex, HSE University, and IST Austria developed "Hogwild! Inference," a protocol enabling multiple Large Language Models (LLMs) to generate text concurrently by sharing a dynamically updated attention Key-Value (KV) cache. This mechanism allows LLMs to self-organize and collaborate in real-time, achieving up to 3.6x speedup and improved accuracy on complex reasoning tasks.

Key insights

• Modern reasoning-capable LLMs can "reason to coordinate" out-of-the-box when provided with a shared, real-time memory, dynamically formulating and adapting collaboration plans without explicit fine-tuning.
• Instant, token-wise synchronization via a shared KV cache is crucial for effective multi-LLM collaboration, outperforming methods with step-wise synchronization or independent generation by enabling immediate awareness of partial thoughts.
• Efficient shared memory access can be achieved through RoPE-based KV shifting, which significantly reduces the computational overhead of concurrent attention across distinct token streams.

Summary by alphaXiv.