Reproduce “To See or To Read: User Behavior Reasoning in Multimodal LLMs” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Tianning Dong, Luyi Ma, Varun Vasudevan, Jason Cho, Sushant Kumar, Kannan Achan

Researchers from Walmart Global Tech demonstrated that representing sequential user behavior as images, specifically scatter plots or flowcharts, for multimodal large language models (MLLMs) improved next-purchase prediction accuracy by up to 87.5% and similarity scores by up to 33.9% compared to text-based inputs, often without increasing computational cost.

Key insights

  • Visual representations, such as scatter plots and flowcharts, enable MLLMs to reason more effectively about sequential user behavior compared to textual inputs.
  • The performance gains from visual modalities often do not incur additional computational cost, particularly for GPT models regarding token usage.
  • Different MLLM architectures exhibit varied preferences for specific visual representation types (e.g., scatter plots vs. flowcharts).
  • While visual inputs improve prediction accuracy, the qualitative aspects of MLLM-generated textual explanations do not necessarily show proportional improvement.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.