Reproduce “FM2DS: Few-Shot Multimodal Multihop Data Synthesis with Knowledge Distillation for Question Answering” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Amirhossein Abaskohi, Spandana Gella, Giuseppe Carenini, Issam H. Laradji

The FM²DS framework synthesizes high-quality, complex multimodal multihop question answering (MMQA) data from long documents using large vision-language models and knowledge distillation. Models fine-tuned on this synthetic data consistently surpass those trained on human-annotated datasets, demonstrating improved performance, generalizability, and reduced hallucinations while offering a cost-effective solution for data scarcity.

Key insights

  • Meticulously generated and validated synthetic data can enable models to outperform those trained on human-annotated datasets, particularly for complex multimodal multihop reasoning tasks.
  • Strategic, multi-stage validation is crucial for ensuring the quality, factual accuracy, and specific reasoning requirements (multihop, multimodal) of synthetically generated data.
  • Cost-effective data synthesis facilitates efficient domain adaptation for smaller LVLMs, allowing them to achieve competitive or superior performance on specialized tasks compared to larger, general-purpose models or limited human-annotated data.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.