Reproduce “Multi-Branch Transformer” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Gaotang Li, Lingjie Chen, Chunyuan Deng, Zhiwei Xu, Hongzheng Chen, Christopher De Sa, Jingrui He, Yingheng Wang and 1 more
The Multi-Branch Transformer (MBT) sends tokens through shared blocks, parallel paths of full Transformer blocks, then fused shared processing. Branches at each depth can share keys-and-values memory while retaining separate queries. Across four tested model scales, branching lowers validation loss; decoding reaches up to 1.50× throughput in specified H200 settings. These results are limited to tested architectures and serving setup.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.