Reproduce “Execution-State Capsules: Graph-Bound Execution-State Checkpoint and Restore for Low-Latency, Small-Batch, On-Device Physical-AI Serving” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Liang Su
FlashRT saves complete inference state in capsules for interactive serving, rather than only token-addressed KV cache, the attention keys and values for prior tokens. Capsules preserve recurrent and convolution state in hybrid models, then resume computation on a new suffix. On RTX 5090, a 16,384-token prefix reduced TTFT from 1,541 ms to 57 ms; this is a first-token latency result, not evidence about throughput or robot performance.
Key insights
- For hybrid LLMs, capturing and restoring the *entire* execution state, not just the KV cache, is crucial for correct and exact reuse, as KV-only restoration leads to immediate output divergence.
- Graph-bound execution-state capsules offer a distinct design point for AI serving, optimizing for extreme low latency in single-stream, on-device scenarios, complementing existing throughput-oriented systems.
- Explicitly managing and pinning full execution states via capsules provides more consistent low-latency performance under working set pressure compared to automatic cache eviction policies, which can suffer from cache collapse.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.