Reproduce “Folding Language Models into Themselves” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Ziyang Zhang, Yubin Jing, Yuyao Li, Haofan Wang, Yichen Gong

Concertina compresses a pretrained language model by sharing weights across neighboring layers while preserving their sequence of computation. It runs each shared block once per original layer and adds small, token-conditioned corrections, then distills the student from the teacher. On Qwen3-4B, a 2.27B-parameter model averages 84% of the teacher’s scores across eight benchmarks, without reducing full-depth forward FLOPs or KV-cache size.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.