Reproduce “Kernel Dependence Network” on OpenResearch

About this paper

Chieh Wu, Aria Masoomi, Arthur Gretton, Jennifer Dy

KNet trains classifiers without backpropagation by building networks layer by layer. At each layer, it uses labels and a Gaussian-kernel objective with iterative spectral updates to bring same-class representations together and separate different-class ones. On the face dataset, KNet reached 0.99 ± 0.01 test accuracy in 0.78 ± 0.08 seconds, versus matched MLPs; performance was low on Random and Adversarial data.

Key insights

• Constraining weights to the Stiefel Manifold allows for spectral optimization, which intrinsically determines each layer's width and the network's overall depth based on spectral rank and HSIC convergence.
• The HSIC objective incorporates an implicit regularization mechanism, contributing to the model's generalization capabilities without the need for explicit regularizers.
• Theoretical guarantees confirm the existence of a 'Kernel Sequence' (composition of layers) that converges strictly monotonically to the global optimum as layers are added.

Summary by alphaXiv.