Reproduce “Deterministic Inference across Tensor Parallel Sizes That Eliminates Training-Inference Mismatch” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ziyang Zhang, Xinheng Ding, Jiayi Yuan, Rixin Liu, Huizi Mao, Jiarong Xing, Zirui Liu
A framework introducing Tree-Based Invariant Kernels (TBIK) ensures bit-wise identical LLM inference outputs across varying Tensor Parallel (TP) sizes by enforcing a fixed floating-point reduction order. This eliminates the "training-inference mismatch" in reinforcement learning pipelines, leading to improved convergence and performance in on-policy RL.
Key insights
- Tensor Parallelism (TP) introduces non-determinism in row-parallel layers by altering the global accumulation order during inter-GPU all-reduce operations, a challenge not met by prior batch-invariant methods.
- A unified, fixed binary-tree reduction topology, spanning both intra-GPU computations and inter-GPU collective operations, can guarantee bit-wise identical results regardless of TP configuration.
- Eliminating the numerical discrepancy between RL rollout and training environments effectively transforms implicit off-policy updates back into true on-policy updates, improving learning stability and outcome.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.