Reproduce “Zipbench: Low-Cost Framework for Compressing Comprehensive Benchmarks of Large Language Models” on OpenResearch

About this paper

Zhongzhan Huang, Junxin Li, Guoming Ling, Yupei Lin, Shanshan Zhong, Hefeng Wu

ZipBench compresses broad LLM benchmark suites using only six full-benchmark evaluations, supplemented by pseudo-model response logs generated from anchor-model voting. Across more than 100 text, multimodal, and agent benchmark configurations, its compact versions reduce item counts by an average of 57.4% to 77.5% while maintaining mean absolute errors of 0.008–0.015 and Spearman correlations of 0.98–0.99 on held-out models.

Key insights

• Anchor diversity is more important than simply increasing the number of evaluated models: broadly distributed performance levels provide better item-response information than clustered anchors.
• Synthetic response patterns can improve coverage without additional model queries, although excessive pseudo-anchor generation may add redundancy and weaken the contribution of real evaluations.
• Theoretical analysis links score fidelity to within-cluster representation distance and explains why rank preservation is harder when models have very similar benchmark scores.

Summary by alphaXiv.