Reproduce “LASER: Low-Rank Activation SVD for Efficient Recursion” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ege Çakar, Ketan Ali Raghu, Lia Zheng
Harvard University researchers developed LASER, a dynamic low-rank compression framework that reduces activation memory by approximately 60% in Tiny Recursive Models (TRMs) during training, without degrading model performance. The work demonstrates that TRM activations, particularly in high-dimensional layers, naturally reside within a low-dimensional linear subspace, revealing geometric properties of implicit reasoning.
Key insights
- TRM activations exhibit a low-rank linear geometry, with rapid spectral decay across recursive steps, implying that implicit reasoning trajectories occupy a constrained, low-dimensional subspace.
- The low-rank structure is genuinely linear, not a locally flat non-linear manifold, suggesting that recursive weight-sharing and architectural elements constrain computation to dominant eigendirections.
- TRM activations possess well-behaved statistical properties, such as near-Gaussian distributions without heavy tails, making them highly compatible with low-rank compression and subsequent INT8 quantization.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.