Reproduce “AutoM3L: An Automated Multimodal Machine Learning Framework with Large Language Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Daqin Luo, Chengjian Feng, Yuxuan Nong, Yiqing Shen

The AutoM3L framework introduces an automated multimodal machine learning approach that leverages Large Language Models (LLMs) as intelligent controllers to construct and optimize training pipelines. It significantly enhances user-friendliness and automation, outperforming existing AutoML methods like AutoGluon in task execution time and user action accuracy while achieving competitive or superior predictive performance on various multimodal and unimodal datasets.

Key insights

  • LLMs can act as effective intelligent controllers for automating complex machine learning pipeline construction, extending beyond rule-based systems to dynamic, interactive orchestration.
  • Representing multimodal data in structured tables is highly effective for LLM-driven analysis and inference of modalities and features.
  • Automated feature engineering and hyperparameter optimization, guided by LLMs, can consistently mitigate noise and effectively find optimal configurations for diverse datasets.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.