Reproduce “lm-Meter: Unveiling Runtime Inference Latency for On-Device Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Haoxin Wang, Xiaolong Tu, Hongyu Ke, Huirong Chai, Dawei Chen, Kyungtae Han
The authors introduce lm-Meter, an on-device profiler that tracks LLM inference phases and individual GPU kernels using lightweight timers and event callbacks. On a Pixel 8 Pro, phase-latency measurements agree with an external reference by at least 99.99% across five models; under Powersave, throughput falls 2.58% in prefill and 0.99% in decode, in that tested setting.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.