Reproduce “TerMeZO: Ternary Sparse Zeroth-Order Optimization for Fine-tuning BitNet Models at the Edge” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Houssem Sifaou, Prabodh Katti, Bipin Rajendran, Osvaldo Simeone
TerMeZO adapts ternary BitNet models by using zeroth-order updates on latent weights near ternary quantization boundaries, while also updating other full-precision parameters. This boundary-based selection uses no first-order gradients to build its mask. On GSM8K, Falcon-E-1B-Base reaches 31.8% accuracy, near MeZO’s 32.6%; a measured H100 setup uses 2.08 GB versus 6.28 GB for MeZO, not validated on edge hardware.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.