Reproduce “OPEN-1B: A Fully Auditable Training Run” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
John Donaghy, Brian Wilcox, Oğuzhan Ersoy, Shikhar Rastogi, Adam St Arnaud, Alexey Titov, Jordan Greenberg, Ben Fielding and 1 more
Open-1B makes language-model training steps replayable across hardware so auditors can compare results with published hashes. It standardizes numerical operations, data order, and gradient reductions, and releases a 400-billion-token run with replay tools and intermediate checkpoints. The paper proposes collective stepwise auditing, but reports no completed independent audit of the full run; RepOps, its core reproducibility library, is proprietary.
Key insights
- Exact replay can make the computation behind a checkpoint testable across hardware, but the guarantee depends on verifying the relevant steps and trusting the integrity of the released reference states and software.
- The training run exposed stability challenges with int8 quantization-aware training, including a query-branch gradient issue and quantizer-range collapse; recalculating quantizer step sizes from current weights addressed the latter in reported tests.
- Reproducibility has a marked systems cost: fixed-order operations and state hashing reduce throughput, while deterministic replay supports accountability rather than proving that the training data are unbiased or benign.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.