Reproduce “Signature-Kernel Based Evaluation Metrics for Robust Probabilistic and Tail-Event Forecasting” on OpenResearch

About this paper

Benjamin R. Redhead, Thomas L. Lee, Peng Gu, Víctor Elvira, Amos Storkey

New kernel-based evaluation metrics, Signature Maximum Mean Discrepancy (Sig-MMD) and Censored Signature Maximum Mean Discrepancy (CSig-MMD), were developed at the University of Edinburgh to assess probabilistic time series forecasts. Sig-MMD captures complex inter-variate and inter-temporal dependencies, while CSig-MMD prioritizes tail-event prediction accuracy, both demonstrating improved sensitivity to forecast errors in dependencies and extreme events compared to standard metrics.

Key insights

• Traditional metrics like CRPS and Energy Score exhibit limited sensitivity to errors in inter-variate and inter-temporal dependencies, failing to accurately penalize forecasters that miss complex correlation structures.
• CSig-MMD effectively isolates and evaluates a forecaster's performance on tail events while remaining robust to errors in the distribution's body, addressing a critical need for risk-aware forecasting.
• The type of model output head significantly impacts tail-event forecasting ability, with parametric output heads sometimes outperforming quantile regression heads in extrapolating extreme tails for certain data.

Summary by alphaXiv.