Reproduce “Local proxies for memory-efficient training” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Wu S. Zihan, Brian A. Yuan, Guillaume Bellec

ProxyProp trains a small proxy to predict a network’s downstream output, then uses the task-loss gradient through that proxy to update one block at a time, reducing activation storage compared with backpropagation. In an 8B LLaMA memory test using artificial data, four-block ProxyProp fit sequences over 50% longer than backpropagation; this measures memory capacity, not task performance on longer sequences.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.