Reproduce “Connecting Giants: Synergistic Knowledge Transfer of Large Multimodal Models for Few-Shot Learning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Hao Tang, Shengfeng He, Jing Qin

The SYNTRANS framework introduces a systematic approach to enhance few-shot learning by extracting and integrating diverse knowledge from large multimodal models (LMMs). It achieves competitive to superior accuracy on benchmark datasets, such as 81.30% in 5-way 1-shot MiniImageNet with a ViT-S backbone, by combining semantic-aligned visual information and rich semantic descriptors.

Key insights

  • Rich, context-aware semantic descriptions generated by Large Language Models and encoded by Visual-Language Models are critical for superior few-shot learning performance compared to simpler semantic sources.
  • Effective knowledge distillation from large multimodal models can imbue lightweight, frozen few-shot vision encoders with powerful vision-semantic alignment capabilities.
  • Adaptively fusing complementary visual and semantic knowledge results in more discriminative and compact classifier weights, significantly improving the recognition of novel categories, especially in extreme low-data settings.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.