Reproduce “On the Opportunities and Challenges of Prefill-to-Autoregressive Cross-Model Transfer” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhenyi Shen, Junru Lu, Wei Liu, Jiangnan Ye, Jiazheng Li, Di Yin, Xing Sun, Tameem Adel and 3 more
PLatour uses prefill-to-autoregressive cross-model transfer to let a larger language model pass internal representations to a smaller receiver, which generates the answer. During prompt processing, trained bridges add projected sender states to the receiver; generation then proceeds without the sender. On GSM8K, it reaches 89.35% avg@2 versus 75.09% for C2C, but its PACT-Score is only +0.23 points, showing that gains beyond tested controls vary by task.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.