Reproduce “ADEPT: Adaptive Dynamic Early-Exit Process for Transformers” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Sangmin Yoo, Srikanth Malla, Chiho Choi, Wei D. Lu, Joon Hee Choi

ADEPT introduces a framework for Transformer models that enables dynamic token-level early exits across both prefill and generation phases, overcoming the Key-Value cache bottleneck with a lightweight auxiliary module. It achieves up to 25% computational savings and 4x acceleration in inference, while improving language modeling performance by up to 45% and requiring an order of magnitude fewer trainable parameters for its early-exit mechanism compared to prior methods.

Key insights

  • Leveraging the observed state convergence in Transformers allows for the prediction of a token's final hidden state, even if it exits intermediate layers.
  • Decoupling Key-Value (KV) cache generation through a predicted final hidden state is crucial for overcoming sequential dependencies and enabling effective token-level early exits.
  • A lightweight, shared auxiliary module and a similarity-based criterion can effectively guide dynamic token-level computation with minimal additional training or inference overhead.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.