Reproduce “VL-RouterBench: A Benchmark for Vision-Language Model Routing” on OpenResearch

About this paper

Zhehao Huang, Baijiong Lin, Jingyuan Zhang, Jingying Wang, Yuhang Liu, Ning Lu, Tao Li, Xiaolin Huang

Researchers from Shanghai Jiao Tong University and The Hong Kong University of Science and Technology introduced VL-RouterBench, a large-scale benchmark for evaluating vision-language model routing strategies, comprising 14 datasets and 17 VLM candidates. The benchmark demonstrates that learned routers consistently surpass single VLMs in accuracy-cost trade-offs, providing up to a 5x cost reduction for similar accuracy compared to the strongest single model.

Key insights

• The substantial performance and cost gap between an ideal Oracle router (95.60% accuracy, $0.37/10K samples) and the best single VLM (78.01% accuracy, $2.72/10K samples) confirms the strong potential for VLM routing.
• Effective VLM routing necessitates multimodal input representations, with a combination of robust text and visual encoders (e.g., BGE-M3 and SigLIP-L-16) and simple fusion methods like normalized concatenation proving beneficial.
• Current learned routers struggle with "hard-route" samples where few models can correctly respond, indicating a need for architectures that can extract finer multimodal discriminative signals.

Summary by alphaXiv.