Reproduce “LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun and 5 more

LayerSkip trains models to let early transformer layers draft tokens, then uses deeper layers to verify and correct them, reusing the same model’s weights and cache. On TOPv2, a 1.5B-parameter model achieved a 2.0× speedup at exit layer E=6: 18 ms per token versus 36 ms, with exact match falling from 85.9% to 82.9%. This H100 result does not mean fewer output tokens.

Key insights

  • Large Language Models can be made robust to early exits and support self-speculative decoding through a novel training recipe that combines non-uniform layer dropout and early exit loss.
  • A single LLM can efficiently perform self-speculative decoding by using its early layers for drafting and later layers for verification, significantly benefiting from shared KV and exit query cache reuse.
  • Standard LLM pretraining can cause intermediate layers to diverge in perplexity, making them less suitable for early exit; LayerSkip's training counteracts this, improving the accuracy of earlier layers while maintaining final layer performance.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.