Reproduce “CF-VLM:CounterFactual Vision-Language Fine-tuning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jusheng Zhang, Kaitong Cai, Yijia Fan, Jian Wang, Keze Wang
CF-VLM improves vision-language models' causal reasoning and fine-grained semantic discrimination by employing a multi-objective fine-tuning framework that leverages strategically generated counterfactual image-text pairs. The approach leads to up to 4.9 percentage point improvements on compositional reasoning tasks and helps reduce visual hallucinations.
Key insights
- Systematically training VLMs with causally-motivated counterfactual samples fosters a deeper understanding of semantic shifts beyond superficial correlations.
- Multi-modal counterfactual generation, involving joint image and text edits, is more effective than single-modality interventions for enhancing causal reasoning.
- Combining foundational alignment with specific counterfactual scenario discrimination and fine-grained causal discrimination objectives synergistically improves VLM performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.