Reproduce “VLLaVO: Mitigating Visual Gap through LLMs” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Shuhao Chen, Yulong Zhang, Weisen Jiang, Jiangang Lu, Yu Zhang

VLLaVO addresses visual domain shifts in cross-domain learning by converting images into textual descriptions via Vision-Language Models (VLMs), then fine-tuning a Large Language Model (LLM) for classification. This approach establishes new state-of-the-art accuracy on several Domain Generalization and Unsupervised Domain Adaptation benchmarks, including PACS, OfficeHome, and DomainNet.

Key insights

  • Textual descriptions, when processed by an LLM, can inherently contain more domain-invariant information than raw visual features, effectively bridging the visual domain gap.
  • Fine-tuning the LLM with a specific instruction template significantly enhances its ability to perform visual classification tasks from textual inputs and improves its generalization.
  • The fine-tuned LLM prioritizes category-relevant words in the textual descriptions, indicating successful guidance to focus on essential information and mitigate irrelevant domain-specific contexts.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.