Reproduce “Adaptive Execution for Out-of-Core LLM Inference” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jiawei Luo, Di Wu, Simon Dobson, Blesson Varghese
AdapLLM enables out-of-core local LLM inference by coordinating layer-wise weight loading with computation. It evicts completed weights, asynchronously prefetches upcoming layers, and adjusts the loading window, precision, and I/O workers to deployment and runtime conditions. Across 16 tested model–device combinations, it outperformed five evaluated baselines, with 2.00× average speedup over default llama.cpp; results are limited to tested settings.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.