Reproduce “(GG) MoE vs. MLP on Tabular Data” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Andrei Chernov
This paper introduces Gumbel-Gating Mixture-of-Experts (GG MoE) models, which incorporate Gumbel-Softmax gating for regularization, to address challenges of deep learning on tabular data. E+GG MoE models, utilizing piecewise-linear embeddings, achieved a lower average performance rank (1.19 " " 0.46) across 38 diverse tabular datasets while demonstrating greater parameter efficiency compared to standard MLPs and MoE approaches.
Key insights
- The Gumbel-Softmax gating in GG MoE provides regularization that prevents degenerate expert utilization.
- Piecewise-linear embeddings are crucial for improving the performance of MoE models on tabular data, especially for MoE and GG MoE.
- MoE and GG MoE models achieve comparable or better performance with significantly fewer parameters than MLPs, enhancing parameter efficiency.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.