Reproduce “LLM-42: Enabling Determinism in LLM Inference with Verified Speculation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Raja Gond, Aditya K Kamath, Ramachandran Ramjee, Ashish Panwar
LLM-42, developed by Microsoft Research and the University of Washington, introduces a verified speculation system to enable deterministic Large Language Model (LLM) inference while preserving the performance benefits of dynamic batching. The system demonstrated up to 41% higher throughput compared to prior deterministic methods for workloads with low deterministic traffic and maintained recomputation costs below 11%.
Key insights
- Token-level inconsistencies from floating-point drift are rare, but quickly amplify into sequence-level divergence if not addressed (Observation 1).
- GPU kernels are typically "position-invariant" and use uniform, shape-consistent reductions (Observation 2), enabling a deterministic verification pass if input shapes are fixed.
- Determinism only requires position-consistent reductions for a given request, rather than batch-invariant computation across all requests, allowing for selective determinism and better performance (Observation 3 & 4).
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.