Reproduce “Herschel: Continuous Optimization of Production LLM Inference through On-Demand Profiling” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Luping Wang, Weigao Chen, Yifei Wu, Yonghe Zhang, Rui Zhang, Wenchao Wu, Jiyu Luo, Haoran Geng and 6 more

Herschel profiles language model services in bounded windows, gathering more evidence when traces cannot explain inefficiencies. It tests workload-matched fixes. Controlled tests report active Hprof-full overhead below 0.5% for TTFT and 7% for TPOT across tested models and workloads. In a production case, supplementary profiling identified synchronous KV-object creation delaying dispatch; an asynchronous fix increased short-context GLM-5.2 prefill throughput 3.3%.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.