Reproduce “Competing Learning in Hybrid Architectures” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xingyu Shen, Yingfa Chen, Ziqing Qiao, Chaojun Xiao, Xu Han, Zhiyuan Liu
BLD gives attention layers an early chance to learn retrieval by freezing recurrent blocks, then unfreezing them for the rest of pretraining. In KDA hybrids from 180M to 1B parameters, it improves retrieval measures; at 510M, Recall rises from 30.50 to 34.70 and RULER from 25.25 to 29.42, while CSR falls from 44.80 to 44.49. Results cover tested settings, not frontier-scale models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.