Reproduce “CCL-Bench 1.0: A Trace-Based Benchmark for LLM Infrastructure” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Eric Ding, Byungsoo Oh, Bhaskar Kataria, Kaiwen Guo, Jelena Gvero, Abhishek Vijaya Kumar, Arjun Devraj, Lindsey Bowen and 3 more
CCL-Bench 1.0 from Cornell University establishes a trace-based benchmark framework for LLM infrastructure, providing fine-grained execution evidence to enable diagnostic insights into performance bottlenecks, post-hoc analysis, and automated optimization. It demonstrates that optimal configurations are often workload and framework-specific, with findings such as a 3x performance difference between frameworks and specific hardware upgrade utilities varying by up to 100x across systems.
Key insights
- Higher compute-communication overlap does not universally reduce LLM training step time; it may reveal inefficient parallelization choices that increase communication volume.
- The utility of doubling interconnect bandwidth in LLM infrastructure is highly dependent on the workload size and underlying hardware architecture, with TPU bandwidth showing up to 100x higher utility than GPU scale-up bandwidth for small workloads.
- Optimal parallelism configurations are highly framework-specific, with a configuration optimal for one training engine potentially being 3x slower on another, even on identical hardware.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.