Reproduce “LANISTR: Multimodal Learning from Structured and Unstructured Data” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Sayna Ebrahimi, Sercan O. Arik, Yihe Dong, Tomas Pfister

Researchers at Google Cloud AI Research developed LANISTR, a multimodal framework that integrates language, image, tabular, and time-series data using a similarity-based masking loss during pretraining. The approach effectively handles pervasive missing modalities and limited labeled data, achieving 87.37% AUROC on the MIMIC-IV dataset and over 75% accuracy on Amazon Product Review tasks with strong out-of-distribution generalization.

Key insights

  • The novel similarity-based multimodal masking loss (`L_SimMMM`) is critical for learning robust cross-modal relationships and effectively utilizing data with high rates of missing modalities during pretraining.
  • Pretraining on partially-available modalities significantly enhances the model's robustness and generalization capabilities, turning data incompleteness into a learning opportunity.
  • Freezing unimodal encoders during fine-tuning allows LANISTR to leverage rich pretrained representations, achieving high performance with as little as 0.01-0.1% of labeled data compared to the pretraining dataset size.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.