Reproduce “A Comprehensive Performance Study of Large Language Models on Novel AI Accelerators” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Murali Emani, Sam Foreman, Varuni Sastry, Zhen Xie, Siddhisanket Raskar, William Arnold, Rajeev Thakur, Venkatram Vishwanath and 1 more

A comprehensive performance study by Argonne National Laboratory researchers evaluated Large Language Models (LLMs) across six distinct AI accelerators and GPUs, quantifying their efficiency and scalability across various workloads including a scientific application. The study found that certain novel accelerators demonstrated performance advantages over traditional GPUs for specific LLM tasks and handling long sequence lengths.

Key insights

  • Novel AI accelerators, specifically SambaNova SN30, Cerebras CS-2, and Graphcore Bow IPUs, can collectively achieve or surpass the performance of a larger number of traditional GPUs for LLM pre-training.
  • Architectures with substantial local memory, such as Cerebras CS-2 and SambaNova SN30, enable fitting and processing LLMs with significantly longer sequence lengths on fewer compute devices.
  • Weight sparsity optimizations on Cerebras CS-2 can lead to considerable throughput improvements (e.g., up to 3.79x for GPT-3 30B at 0.9 sparsity), although this may involve trade-offs in training loss.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.