Reproduce “Together Compiler: Continual Compilation for AI Inference” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Reyna Abhyankar, Dylan Lim, Drew Wadsworth, Venkata Hasith Vattikuti, YuTong Julia Zhang, Jung Hwan Heo, Taeksang Kim, Kesavan Ramakrishnan and 4 more

TKCC optimizes changing inference traffic without restarting the engine for each candidate. It hot-reloads forward passes, screens candidates with a fast tier, checks them at higher fidelity, and uses a cost model to narrow GPU parallelism choices. On MiniMax M3.1 Flash on B200, it reports 1,111 P90 tokens per second per user versus 633 for InferenceX, in a low-concurrency comparison using TP8.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.