Reproduce “OUTLETS: Output-Length Prediction from Speculative Decoding Backbones” on OpenResearch

About this paper

Weihuang Wen, Yingying Liu, Yichuan Liu, Wenqi Zeng, Li Zhou, Chumin Sun, Jie Sun, Tianshu Yu

OUTLETS repurposes the EAGLE-style speculative-decoding backbone of an LLM to predict how many tokens remain in a generation, avoiding a separate proxy model. Across Llama and Qwen models, it generally improves output-length prediction over hidden-state, BERT, bucket-classification, and prompting baselines; when used for scheduling, it raises throughput by 3.3% and cuts short-request P99 latency by 34.8%.

Key insights

• Representations useful for speculative token drafting also contain signals about termination and future generation duration, making one inference component useful for both acceleration and scheduling.
• A trajectory-processing draft decoder is more important than feature fusion alone: removing it causes the largest prediction-accuracy drop, while joint length training does not materially reduce speculative-token acceptance.
• Length-aware scheduling primarily benefits short requests: queue ordering provides most of the tail-latency reduction, whereas load balancing adds throughput gains and helps avoid uneven decode workloads.

Summary by alphaXiv.