Reproduce “KAN versus MLP on Irregular or Noisy Functions” on OpenResearch

About this paper

Chen Zeng, Jiahui Wang, Haoran Shen, Qiao Wang

This paper presents a systematic comparison of Kolmogorov-Arnold Networks (KANs) and Multi-Layer Perceptrons (MLPs) across diverse function types and noise conditions. The study found that KANs generally excel at approximating regular, smooth functions but may struggle with non-differentiable points and discontinuities, where MLPs often perform better, while KANs show advantages for functions with singularities and oscillations given appropriate optimization. Noise impacts KANs more severely than MLPs, particularly for already irregular functions, and the choice of optimizer significantly affects performance.

Key insights

• The optimal neural network architecture (KAN or MLP) depends on the specific characteristics of the target function, with KANs excelling at smooth functions and MLPs often preferred for functions with sharp non-differentiable points or jump discontinuities.
• Kolmogorov-Arnold Networks (KANs) are more susceptible to noise than Multi-Layer Perceptrons (MLPs), particularly when fitting already irregular functions, suggesting a need for careful regularization in noisy real-world data.
• The choice of optimizer significantly influences network performance, especially for KANs; Adam generally achieves lower test loss over more epochs, while L-BFGS offers faster initial convergence but is computationally more expensive.

Summary by alphaXiv.