Reproduce “GHC: Expanding Residual Interfaces in Hyper-Connections” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Tianyu Li, Hongtao Zhang, Dongchen Han, Jingshu Liu, Bojun Wang, Chen Jiang, Yulong Huang, Wei Chen and 3 more

GHC expands read and write interfaces in multi-stream language models by grouping branch computation, letting groups mix residual streams separately while sharing the intermediate width. In a 430M dense model, doubling streams from four to eight reduced validation loss by 0.0195 with GHC versus 0.0028 with IHC. This supports an interface bottleneck, but does not show it is the only limit.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.