Reproduce “Searching for Efficient Linear Layers over a Continuous Space of Structured Matrices” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Andres Potapczynski, Shikai Qiu, Marc Finzi, Christopher Ferri, Zixi Chen, Micah Goldblum, Bayan Bruss, Christopher De Sa and 1 more
Researchers at NYU and collaborating institutions introduced a continuous framework to parameterize and analyze structured matrices using Einstein summations, developing a taxonomy to understand their compute-optimal scaling properties. Their findings led to the design of BTT-MoE, a novel architecture that achieves significantly improved compute efficiency, requiring 5.3 times fewer FLOPs than dense models for equivalent performance.
Key insights
- Parameter sharing (ω > 0) and low-rank (ψ < 1) structures are detrimental to compute-optimal scaling, leading to worse performance laws.
- Within the optimal subspace (ω=0, ψ=1), the compute intensity (ν) has minimal impact on scaling laws, meaning structures with sub-quadratic compute can perform similarly to dense layers.
- The analysis clarifies why previous structured matrices showed advantages in memory/inference-constrained settings but not necessarily in compute-optimal training.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.