Reproduce “SecFormer: Fast and Accurate Privacy-Preserving Inference for Transformer Models via SMPC” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jinglong Luo, Yehong Zhang, Zhuo Zhang, Jiaqi Zhang, Xin Mu, Hui Wang, Yue Yu, Zenglin Xu
SecFormer presents a framework for privacy-preserving inference of Transformer models via Secure Multi-Party Computation (SMPC), achieving fast and accurate results by strategically combining model architectural redesign with novel, efficient SMPC protocols for non-linear operations. It demonstrates significant speedups over protocol-focused baselines and substantial accuracy improvements over model redesign-focused baselines across various tasks.
Key insights
- Accurate computation of the GeLU function is critical for maintaining Transformer model performance, particularly for larger models, challenging the efficacy of aggressive quadratic approximations.
- A synergistic approach that combines specific model redesigns with tailored, optimized SMPC protocols for critical non-linearities can overcome the traditional trade-off between efficiency and accuracy in privacy-preserving inference.
- Advanced numerical techniques like segmented Fourier series for `erf` approximation and Goldschmidt's method with input deflation are highly effective in drastically reducing the computational and communication overheads of complex non-linear operations within SMPC.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.