Reproduce “MicroAdam: Accurate Adaptive Optimization with Low Space Overhead and Provable Convergence” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ionut-Vlad Modoranu, Mher Safaryan, Grigory Malinovsky, Eldar Kurtic, Thomas Robert, Peter Richtarik, Dan Alistarh
MICROADAM is a memory-efficient Adam optimizer that uses Top-K sparsification and a novel compressed error feedback mechanism to drastically reduce optimizer state memory for large deep neural networks. It maintains competitive or superior accuracy on various models, including Llama-2 7B/13B fine-tuning and ResNet pre-training, while providing strong theoretical convergence guarantees.
Key insights
- Error feedback mechanisms, traditionally used in distributed optimization, can be effectively adapted to single-node settings to correct for compression errors introduced within the optimizer.
- For practical memory gains, the error feedback buffer itself must be compressed, which is achieved through low-bit quantization in MICROADAM.
- Adam's optimization trajectory can be maintained or even improved with highly sparse gradient information and compressed states, provided a robust error correction mechanism is in place.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.