Reproduce “Adaptive Inference Batching using Policy Gradients” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ruslan Sharifullin
The paper evaluates a reinforcement learning scheduler for inference batching and routing. It uses queue and GPU state to choose batch size and, across GPUs, destination. In simulated heterogeneous workloads, it routes fast and slow requests separately, earning reward of 910.5 versus 612.8 for Shortest-Queue, a 48% gain. On standard single-GPU serving, reward improves 0.1% over tuned static batching; results are simulations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.