Reproduce “Post-Training Quantization for Video Matting” on OpenResearch

About this paper

Tianrui Zhu, Houyuan Chen, Ruihao Gong, Michele Magno, Haotong Qin, Kai Zhang

ETH Zürich, SenseTime Research, and Nanjing University researchers introduce PTQ4VM, the first systematic post-training quantization framework specifically designed for video matting models, achieving near full-precision performance at 8-bit quantization and maintaining satisfactory quality at ultra-low 4-bit settings with 8x FLOPs reduction on RVM baseline, through a novel three-component approach combining block-wise initial quantization for stable convergence, Global Affine Calibration (GAC) that compensates for statistical distortions from batch normalization folding by learning layer-specific scaling/shift parameters, and Optical Flow Assistance (OFA) that leverages RAFT-computed motion fields as temporal priors to enhance coherence across frames, with GAC demonstrating universal applicability by consistently improving existing PTQ methods like BRECQ and QDrop across standard matting metrics (SAD, MSE, Grad, Conn) and temporal consistency measures (DTSSD) on both VM video dataset and D646 image dataset despite calibration on only 256 images.

Key insights

• Conventional PTQ pipelines overlook cumulative statistical alterations in intermediate activations caused by quantization, making folded batch normalization parameters suboptimal for shifted distributions
• Optical flow can serve as an effective temporal regularization mechanism during quantization calibration, helping preserve motion coherence and semantic understanding in compressed video models
• Block-wise optimization strikes an optimal balance between computational efficiency and capturing local inter-layer dependencies compared to layer-wise or full-network approaches

Summary by alphaXiv.