Reproduce “TapeSAC: Automated, Pareto-Optimal Fine-Grained Memory Management for Large Model Training” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Kaihui Ren, Yifan Zhao, Zejian Wang, Yingtong Xiong, Xiaoling Zhou, Daohong Xu, Chenhao Guan, Zhikang Mo and 6 more
TapeSAC helps train large models within GPU memory limits by recording operations and rebuilding only values requested during backward, rather than rerunning whole blocks. Its planner selects values to save using measured replay costs and a memory constraint. On Qwen3.6-35B-A3B, at a 90% device-memory limit, it reduces step time by 13.9% versus full checkpointing; the comparison uses unequal reserved memory.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.