Reproduce “Efficient Deep Learning Pipelines for Accurate Cost Estimations Over Large Scale Query Workload” on OpenResearch

About this paper

Johan Kok Zhi Kang, Gaurav, Sien Yi Tan, Feng Cheng, Shixuan Sun, Bingsheng He

Researchers from the National University of Singapore and Grab developed Prestroid, an optimized deep learning pipeline that accurately estimates query resource consumption while achieving substantial cost savings on large-scale industrial workloads. The pipeline reduced training costs by up to 13.2x and per-batch memory footprint by 13.5x compared to full-tree models, without compromising predictive accuracy.

Key insights

• Industrial query workloads are significantly more diverse and dynamic than public benchmarks (e.g., TPC-DS/H), posing unique challenges for deep learning model scalability and maintenance.
• Optimizing data input (e.g., reducing padding through sub-tree sampling) and efficient predicate encoding are crucial for making deep learning models economically viable for production-scale query cost estimation.
• Sub-tree processing enables models to incorporate K times more features into dense layers, leading to richer mappings and improved predictive accuracy, especially on diverse datasets.

Summary by alphaXiv.