Reproduce “FP8-RL: A Practical and Stable Low-Precision Stack for LLM Reinforcement Learning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Zhaopeng Qiu, Shuang Yu, Jingqi Zhang, Shuai Zhang, Xue Huang, Jingyi Yang, Junjie Lai

FP8-RL accelerates language-model reinforcement-learning rollouts by refreshing the inference engine with each updated policy, block-quantizing selected weights, and correcting policy mismatch with token-level truncated importance sampling (TIS). In a Qwen3-8B-Base DAPO setup, the combined FP8 configuration made rollout 44% faster than BF16. With TIS at clipping threshold C=2, learning remained comparable in this tested setup, not universally.

Key insights

  • Importance sampling is critical for mitigating training-inference mismatch in FP8 LLM RL, enabling stable training behavior comparable to BF16 baselines.
  • FP8 KV cache quantization yields disproportionately high speedups for memory-bound long-context generation, effectively doubling capacity and reducing request preemptions.
  • End-to-end FP8 (both training and rollout) reduces training-inference mismatch, and specific recipes (e.g., hybrid FP8 for training, BF16 router for MoE) are crucial for stability, especially in complex models.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.