Reproduce “Impacts of floating-point non-associativity on reproducibility for HPC and deep learning applications” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Sanjif Shanmugavelu, Mathieu Taillefumier, Christopher Culver, Oscar Hernandez, Mark Coletti, Ada Sedova

The authors investigate how non-associative floating-point addition affects reproducibility in parallel GPU reductions and deep-learning workflows. They compare non-deterministic accumulation with ordered, deterministic reductions, and test indexed aggregation in GraphSAGE. On Cora, all 1,000 models trained with non-deterministic index_add had different weights despite similar losses; this result does not show different predictions or generalize beyond the tested setup.

Key insights

  • The statistical properties of FPNA-induced variability are not universally Gaussian, with the `atomicAdd`-only method showing non-normal distributions, challenging common assumptions about noise.
  • Achieving software-based determinism for parallel sums and PyTorch operations often incurs performance overheads or requires complex programming, although some deterministic approaches can be competitive.
  • Hardware-based determinism, as demonstrated by the Groq LPU™ accelerator, offers a powerful alternative, delivering both consistent reproducibility and superior performance for deep learning inference.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.