Reproduce “Scaling Inference Prefill with High-Radix Photonic Interconnects” on OpenResearch

About this paper

Arulselvan Madhavan, Peter Carson, Taylor Groves, Thomas Graham

Research from Lightmatter quantifies the performance benefits of 3D-integrated photonic interconnects for large language model inference prefill, demonstrating substantial latency reductions. The study shows these optical interconnects enable larger, higher-bandwidth scale-up pods, yielding 2.1-8.5x speedups over electrical systems, especially for communication-intensive large context window workloads.

Key insights

• Photonic interconnects deliver their most substantial gains in communication-bound scenarios, especially when workloads require scaling beyond the single-rack limits of electrical interconnects.
• The ability of optical interconnects to form homogeneous ultra-high-bandwidth scale-up pods of hundreds to thousands of devices significantly mitigates the communication penalties associated with slower cross-rack fabrics.
• While optical interconnects greatly improve prefill latency and input throughput, comprehensive end-to-end LLM serving performance also requires co-designing sufficient memory-bandwidth-bound decode capacity.

Summary by alphaXiv.