Reproduce “BiBench: Benchmarking and Analyzing Network Binarization” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Haotong Qin, Mingyuan Zhang, Yifu Ding, Aoyu Li, Zhongang Cai, Ziwei Liu, Fisher Yu, Xianglong Liu
BiBench established a comprehensive benchmark for network binarization, evaluating 8 algorithms across diverse tasks, architectures, and hardware to analyze accuracy and efficiency. The study revealed varying performance across modalities, challenges with binarizing Transformers, comparable corruption robustness, and identified key elements for practical binarization algorithm design.
Key insights
- Binarized operator design, particularly soft gradient approximation and channel-wise scaling factors, significantly impacts accuracy and deployability.
- Binarizing Transformer-based architectures is challenging due to their reliance on attention mechanisms involving multiplications between binarized activations, leading to significant accuracy drops.
- Binarized networks intrinsically exhibit robustness to data corruptions comparable to full-precision models, likely due to the high discretization of parameters.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.