Reproduce “Building AI Research Teams for Autonomous Discovery of Linear Attention Architectures” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yujian Liu, Li An, Zelin He, Jacob Helwig, Jiabao Ji, Rohit Jain, Siyu Zhu, Shiyu Chang
The authors present a multi-agent system that uses disciplinary perspectives, private histories, shared evidence, and adaptive selection to search linear attention architectures. It discovers Context-First DeltaNet (CFDN), adding a query-independent recurrent vector to GDN2’s query-dependent matrix memory and updating the vector first. At 1.3B parameters, CFDN improves all five category means over GDN2 under matched-FLOP training, but not CSR at 150M.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.