Reproduce “Stage-Replay Divergence Follows the KV Cache: Fixed-Prefix Precision Controls and Bidirectional Cache Transplantation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Alexander Boesgaard Lorup
This research investigates the fidelity of stage-replay methods for large language models, revealing that fresh prefill often fails to reproduce the exact internal state and subsequent output trajectory of a live autoregressive decode in BF16 precision. The study demonstrates that these divergences are causally mediated by the Key/Value cache, with higher numerical precision (FP32) eliminating the observed behavioral differences.
Key insights
- Fresh prefill in BF16 precision leads to significant divergence from live autoregressive decoding in internal state and subsequent greedy trajectories (83% suffix divergence), even with identical discrete token prefixes.
- The complete K/V cache at an intermediate boundary is causally sufficient to determine the subsequent greedy trajectory, as demonstrated by the successful transfer of divergent suffixes through cache transplantation.
- Numerical precision plays a critical moderating role; FP32 consistently eliminates the behavioral differences observed between live and replayed states that are present in BF16.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.