Reproduce “Rethinking Small VLM Quantization: From Component-Wise Analysis to Hardware-Aware Edge Deployment” on OpenResearch

About this paper

Hyeju Shin, Chorwon Kim, Ryangsoo Kim, Hark Yoo, Jaein Kim

Researchers at ETRI conducted a component-wise evaluation of quantization strategies for small Vision-Language Models (sVLMs) on NVIDIA Jetson Orin edge platforms. The study found that quantization sensitivity depends more on architectural paradigms (MoE vs. dense LLMs) than model scale, identified significant latency increases for SigLIP vision encoders under INT8 quantization due to hardware-software friction, and provided insights for hardware-aware mixed-precision deployment.

Key insights

• Quantization sensitivity in sub-3B VLMs is predominantly determined by the LLM backbone's architectural paradigm (MoE vs. Dense), with MoE models demonstrating resilience to INT4 noise.
• SigLIP-based vision encoders exhibit disproportionate latency increases under INT8 quantization on Jetson Orin platforms due to identified hardware-software stack incompatibilities.
• LLM INT4 quantization primarily serves as a VRAM reduction technique on edge devices, consistently increasing time per output token (TPOT) due to dequantization overhead.

Summary by alphaXiv.