Reproduce “Koi: Evidence-Guided, Self-Calibrating Planning for Multi-Cloud, Heterogeneous LLM Inference Fleets” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Mankeerat Sidhu, Gangmuk Lim, Hetarth Chopra, Maximillian Eisenberg, Quan Hao Ng, Aryan Gosaliya, Shu Liu, Luka Govedič and 5 more
Koi coordinates language-model jobs across a shared GPU fleet, using model-proposed plans while deterministic software predicts outcomes, checks feasibility, and selects cluster-wide deployments. Telemetry from those deployments updates scoped hypotheses about runtime mechanisms. In a three-hour benchmark on a heterogeneous 104-GPU cluster, Koi delivered 4,119.51 goodput tokens per second versus OpenEvolve’s 3,497.43, with 20% lower cost per SLO-honoring token; only three real-cluster baselines were tested.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.