Reproduce “Accelerated Learning of High Dimensional Functions with a Tensor-Featured Training Network” on OpenResearch

About this paper

Karl Pierce, Yuehaw Khoo, Haizhao Yang

A "Tensor-Featured training" network is introduced, accelerating the optimization and improving the accuracy of deep neural networks for learning high-dimensional functions by integrating contextual features derived from tensor decomposition. This approach reduced relative L2 error for a 15-dimensional problem from 0.272 to 0.0619 in a 3-step process and achieved 8-34 orders of magnitude memory reduction for tensor decomposition using randomized methods.

Key insights

• Tensor decomposition of DNN outputs into a Canonical Polyadic Decomposition (CPD) acts as a smoothing and regularization mechanism, improving function accuracy by incorporating non-local information.
• Tensor-derived features provide substantial acceleration in DNN training, but their effectiveness depends on the initial accuracy and contextual relevance of the model used to generate the features.
• An iterative, multi-step optimization process, particularly when combining simple Rank-1 features with refined tensor features, consistently reduces error and improves the accuracy of high-dimensional function learning.

Summary by alphaXiv.