Reproduce “Building AI Research Teams for Autonomous Discovery of Linear Attention Architectures” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yujian Liu, Li An, Zelin He, Jacob Helwig, Jiabao Ji, Rohit Jain, Siyu Zhu, Shiyu Chang

The authors present a multi-agent system that uses disciplinary perspectives, private histories, shared evidence, and adaptive selection to search linear attention architectures. It discovers Context-First DeltaNet (CFDN), adding a query-independent recurrent vector to GDN2’s query-dependent matrix memory and updating the vector first. At 1.3B parameters, CFDN improves all five category means over GDN2 under matched-FLOP training, but not CSR at 150M.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.