Reproduce “DEPT: Decoupled Embeddings for Pre-training Language Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Alex Iacob, Lorenzo Sani, Meghdad Kurmanji, William F. Shen, Xinchi Qiu, Dongqi Cai, Yan Gao, Nicholas D. Lane

Researchers at Flower Labs and the University of Cambridge developed DEPT, a pre-training framework that decouples token embeddings from the transformer body. This approach achieves substantial communication and memory savings while improving model quality on heterogeneous datasets, particularly benefiting federated learning environments by enabling vocabulary-agnostic pre-training for billion-scale models.

Key insights

  • The transformer body can learn robust, generalized representations effectively without relying on a rigid, shared global embedding space.
  • Decoupling embeddings allows them to specialize to local data sources, mitigating issues like negative interference and vocabulary dilution arising from diverse datasets.
  • Aggregating only the transformer body parameters (or trimmed embeddings) offers significant communication and memory efficiencies, which are crucial for scaling language model pre-training, especially in federated learning environments.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.