Reproduce “Request-Adaptive Execution for Efficient Hybrid LLM Inference” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Fahao Chen, Xuzhe Yan, Qi Zhang, Peng Li, Dongxiao Yu, Song Guo
Flex lets each request choose Attention or SSM branches block by block from its prompt, then holds that route fixed during decoding. Its runtime batches divergent branch work and reuses caches only when text prefixes and layer-choice paths match. At 14.5B, Flex reaches 57.7% ARC-Challenge accuracy versus 52.6% for dataset-level routing under matched-cost candidates, showing an advantage in that comparison.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.