Reproduce “Decoupled Mixture-of-Experts for Parametric Knowledge Injection” on OpenResearch

About this paper

Baoqing Yue, Weihang Su, Qingyao Ai, Yichen Tang, Changyue Wang, Jiacheng Kang, Jingtao Zhan, Yiqun Liu

Researchers at Tsinghua University developed Decoupled Mixture-of-Experts (DMoE), an architecture that injects parametric knowledge into large language models using lightweight, conditionally activated LoRA experts. This approach enables deep knowledge integration while preserving KV-cache reuse, achieving competitive answer quality on knowledge-intensive tasks with approximately 3x faster inference and 1.6x–1.9x less GPU memory compared to dynamic Retrieval-Augmented Generation methods.

Key insights

• Decoupling experts and the router from the base model allows for highly modular knowledge injection, enabling independent updates and additions of specific knowledge units without disturbing existing capabilities.
• Placing knowledge experts only at the final-layer FFN is crucial for maintaining KV-cache reuse, significantly reducing inference latency and memory consumption compared to methods that invalidate the cache.
• Token Uncertainty (TU) serves as an effective signal for conditionally triggering expert activation, ensuring that external knowledge is engaged only when the base model is uncertain or lacks sufficient internal knowledge.

Summary by alphaXiv.