Reproduce “MoIN: Mixture of Introvert Experts to Upcycle an LLM” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ajinkya Tejankar, KL Navaneet, Ujjawal Panchal, Kossar Pourahmadi, Hamed Pirsiavash
MOIN, a framework from the University of California, Davis, introduces a modular architecture that extends a base LLM with specialized 'introvert experts' for specific topics, significantly reducing the computational burden of improving existing models. This approach allows a TinyLlama-based model to achieve perplexity comparable to a TinyLlama trained on 500 billion more tokens and perform comparably on downstream tasks.
Key insights
- Sequence-level routing to a single specialized expert per query, rather than token-level routing, enables substantial memory efficiency by only requiring one expert in GPU memory at a time.
- Leveraging lightweight LoRA adapters for each topic-specific expert allows for efficient fine-tuning, reduces training costs, and facilitates modularity for continual learning.
- The quality of topic clustering from pre-training data is crucial for creating specialized experts, with both UMAP/HDBSCAN and K-means clustering yielding semantically meaningful topics.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.