Reproduce “TinyCeNN-LM: Quality-Gated Conversion of Pretrained Attention with CeNN-Inspired Cellular-Recurrent Layers” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Kabeh Mohsenzadegan, Vahid Tavakkoli, Kyandoghere Kyamakya
The authors present TinyCeNN-LM, a method for testing proposed attention replacements in pretrained language models before committing them. It trains one candidate layer at a time and accepts it only when representation-fidelity and language-model-loss thresholds pass, otherwise restoring native attention. SmolLM2 layer 3 passed loss limits but failed representation checks, showing loss alone can miss compatibility drift, not downstream failure.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.