Reproduce “GraphNet: A Large-Scale Computational Graph Dataset for Tensor Compiler Research” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xinqi Li, Yiqun Liu, Shan Jiang, Enrong Zheng, Huaijin Zheng, Wenhao Dai, Haodong Deng, Dianhai Yu and 1 more
Researchers from Baidu Inc.'s PaddlePaddle Team introduced GraphNet, a dataset comprising over 2.7K real-world deep learning computational graphs from diverse PaddlePaddle and PyTorch models, to enable standardized evaluation of tensor compilers. The work also defines novel Speedup Score (S_t) and Error-aware Speedup Score (ES_t) metrics that assess performance based on speedup, numerical correctness, and specific failure modes.
Key insights
- The Error-aware Speedup Score (ES_t) effectively diagnoses different failure types (accuracy errors, runtime crashes, compilation failures), offering more actionable insights than simple aggregated speedup metrics.
- Benchmarking with GraphNet revealed that default compilers (CINN and TorchInductor) face distinct challenges, including numerical precision issues for NLP models, accuracy violations for PaddlePaddle-CV, and widespread compilation failures and negative optimizations for PyTorch-CV.
- Large-scale, authentic computational graph datasets are essential for systematically evaluating current tensor compilers and for training and evaluating future AI-driven compiler optimization methods.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.