Reproduce “CoRun: Padding is Simple and Efficient for Deterministic LLM Inference” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shiju Zhao, Jiacheng Yang, Qihang Chen, Junhao Hu, Jiaqi Zheng, Guihai Chen, Xusheng Chen
CORUN, developed by researchers from Nanjing University and Tencent, introduces a system for deterministic Large Language Model (LLM) inference that achieves output consistency by controlling execution shapes through padding. This approach leverages faster position-invariant GPU kernels to deliver throughput improvements of 15% to 324% compared to batch-invariant methods, effectively doubling aggregate throughput across various models and workloads.
Key insights
- Most optimized GPU kernels, while not fully 'batch-invariant,' are 'position-invariant,' meaning their results are consistent for a fixed batch shape, creating a new avenue for deterministic optimization.
- Deterministic LLM inference can be achieved by strategically controlling the execution shapes presented to kernels through padding and isolation, rather than globally enforcing batch-invariant kernel implementations.
- The separation of prefill (isolated) and decode (fixed-shape batched with CUDA Graphs) stages allows for optimized, deterministic execution without the performance overhead of traditional batch-invariant approaches.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.