Reproduce “Duo-LLM: A Framework for Studying Adaptive Computation in Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Keivan Alizadeh, Iman Mirzadeh, Hooman Shahrokhi, Dmitry Belenko, Frank Sun, Minsik Cho, Mohammad Hossein Sekhavat, Moin Nabi and 1 more
Duo-LLM studies adaptive computation by routing tokens through small or large feed-forward modules, or skipping layers, with shared attention. It compares learned routing with an exhaustive oracle that selects routes using ground-truth next-token labels. On held-out C4, one large-module layer per token can achieve lower perplexity than using large modules in all 12 layers; this oracle result does not demonstrate deployable speed gains.
Key insights
- Token difficulty is relative; tokens benefiting most from additional computation are not necessarily those with the highest loss values.
- Optimal allocation of computational resources (big modules) is budget-dependent, shifting from later to earlier layers as the budget increases.
- Selectively placing a few big modules in specific layers often outperforms using big modules throughout all layers, challenging conventional intuition.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.