Reproduce “TabLoRA: Parameter-Efficient Low-Rank Ensemble Learning for Large-Scale Tabular Data” on OpenResearch

About this paper

Jiaqi Luo, Shixin Xu

TabLoRA introduces a parameter-efficient neural ensemble framework for large-scale tabular data, utilizing a shared backbone augmented with predictor-specific low-rank adaptations and feature transformations. This approach achieves competitive predictive performance comparable to full deep ensembles while significantly reducing parameter count and maintaining stable memory usage, successfully operating on all large-scale datasets where other deep learning models frequently failed due to memory constraints.

Key insights

• The concept of low-rank adaptation, typically for fine-tuning pre-trained models, can be effectively repurposed for parameter-efficient ensemble learning from scratch in tabular data.
• Decomposing a full deep ensemble into a shared computational core augmented with small, predictor-specific low-rank perturbations allows for high expressive power without the linear parameter growth of traditional ensembles.
• Introducing lightweight feature transformations at the input level enhances diversity among ensemble members, contributing to improved predictive performance with minimal overhead.

Summary by alphaXiv.