Reproduce “MLPerf Inference Benchmark” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Vijay Janapa Reddi, Christine Cheng, David Kanter, Peter Mattson, Guenther Schmuelling, Carole-Jean Wu, Brian Anderson, Maximilien Breughe and 39 more
MLPerf Inference v0.5 defines a shared benchmark for comparing inference systems under different request patterns, pairing reference workloads and quality targets with four scenarios. Its LoadGen measures performance under single-stream, multistream, server, and offline traffic. Across five systems with neural machine translation results, server throughput was 39–55% below offline throughput, a finding limited to these benchmark scenarios.
Key insights
- Performance evaluation must account for realistic end-user scenarios, as system behavior varies significantly under different latency and throughput requirements.
- Standardizing quality targets (e.g., accuracy within 1-2% of FP32) alongside performance metrics is essential to prevent misleading high-performance claims at unacceptable quality levels.
- A comprehensive, auditable benchmarking framework that balances strict comparability with flexibility for innovation is critical for a rapidly evolving field.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.