Reproduce “QCFE: An efficient Feature engineering for query cost estimation” on OpenResearch

About this paper

Yu Yan, Hongzhi Wang, Junfang Huang, Dake Zhong, Man Yang, Kaixin Zhang, Tao Yu, Tianqing Wan

QCFE introduces an efficient feature engineering framework for AI-driven query cost estimation, integrating previously ignored database variables via a feature snapshot and pruning ineffective features using a difference-propagation method. This approach consistently improved prediction accuracy (e.g., reducing job-light's q-error from 1.316 to 1.172) and reduced training time by up to 37.3% across various benchmarks, while also demonstrating strong transferability to new hardware environments.

Key insights

• Crucial database variables beyond query plans and table statistics, such as hardware and database configurations, significantly influence query costs and must be integrated into AI models for higher accuracy.
• Optimizing input features through efficient feature engineering can more effectively enhance both accuracy and efficiency of AI-driven cost estimators than solely relying on increasingly complex neural network architectures.
• Simplified query templates can efficiently generate labeled data to estimate the feature snapshot, providing an effective and time-saving way to represent the influence of ignored variables.
• Difference-propagation offers a robust method for feature importance calculation that is superior to gradient-based or greedy approaches for database features, particularly when dealing with discrete, one-hot encoded inputs.

Summary by alphaXiv.