Reproduce “Muddling Label Regularization: Deep Learning for Tabular Datasets” on OpenResearch

About this paper

Karim Lounici, Katia Meziani, Benjamin Riu

A new deep learning approach, Muddling Label Regularization (MLR), enables standard Feed-Forward Neural Networks (FFNNs) to effectively learn from tabular datasets. This method, leveraging a novel loss function with Ridge regularization, random label permutations, and structured dithering, consistently outperformed "gold standard" ensemble methods like GBDT and Random Forests in regression tasks across 16 diverse datasets, achieving a higher mean R^2 of 0.744.

Key insights

• Applying Ridge regularization to the last hidden layer, coupled with a wide FFNN architecture, is foundational for meaningful learning on tabular datasets.
• Actively penalizing a model's ability to fit randomly permuted labels (memorization) effectively forces it to learn true data patterns and improves generalization.
• The combination of structured regularization and loss function modifications allows standard, simple FFNNs to overcome their traditional weaknesses on tabular data.

Summary by alphaXiv.