Reproduce “COMET: A Novel Memory-Efficient Deep Learning Training Framework by Using Error-Bounded Lossy Compression” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Sian Jin, Chengming Zhang, Xintong Jiang, Yunhe Feng, Hui Guan, Guanpeng Li, Shuaiwen Leon Song, Dingwen Tao
A deep learning training framework, developed by researchers primarily at Washington State University, utilizes error-bounded lossy compression of intermediate activation data to reduce GPU memory consumption. This framework achieves up to 13.5x memory reduction and up to 2x end-to-end training speedup while preserving validation accuracy.
Key insights
- Compression errors in activation data, when uniformly distributed within a controlled bound, propagate to gradients following a normal distribution whose standard deviation can be precisely estimated.
- Maintaining the standard deviation of gradient errors within 1% of the average momentum value is crucial for preserving training stability and final model accuracy.
- Precisely handling continuous zeros during lossy decompression is vital; reconstructing zeros as small non-zero values (within the error bound) can lead to gradient explosion and untrainable models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.