Reproduce “SPlit: An Optimal Method for Data Splitting” on OpenResearch

About this paper

V. Roshan Joseph, Akhil Vakayil

Researchers from Georgia Institute of Technology developed SPlit, an optimal data splitting method grounded in Support Points and energy distance, which creates training and testing sets that closely mirror the full data distribution. This approach demonstrably reduces bias and variability in generalization error estimation for both regression and classification models, offering superior average and worst-case performance compared to random subsampling and existing deterministic methods.

Key insights

• Support Points, initially used for continuous distribution representation, can be adapted to efficiently select optimal discrete subsets from a dataset, offering Quasi-Monte Carlo properties for improved estimation efficiency.
• The representativeness of data splits for categorical variables is significantly influenced by the choice of dummy coding, with Helmert coding outperforming others by yielding tighter clusters and greater separation distances.
• An optimal data splitting method, by accurately reflecting the underlying data distribution, can substantially reduce bias and variability in generalization error estimates compared to random or other deterministic splitting approaches.

Summary by alphaXiv.