Reproduce “All You May Need for VQA are Image Captions” on OpenResearch

About this paper

Soravit Changpinyo, Doron Kukliansky, Idan Szpektor, Xi Chen, Nan Ding, Radu Soricut

Google Research developed VQ²A, an automated method that generates extensive Visual Question Answering (VQA) training data by transforming image captions into question-answer pairs using neural models for question generation and validation. This approach establishes new state-of-the-art zero-shot accuracy on VQA2.0 and GQA, with improvements of +13.2% and +17.6% respectively over prior methods, and substantially boosts performance when used for pre-training on OKVQA.

Key insights

• Automatically generated VQA data, even from noisier web-scale captions (Conceptual Captions), can train VQA models to achieve new state-of-the-art zero-shot performance on established benchmarks.
• Incorporating a diverse range of answer types (beyond just noun phrases) during automatic data generation is crucial for high VQA model performance.
• Models trained on VQ²A data exhibit greater robustness and generalize better to out-of-domain evaluation sets compared to models trained on manually curated VQA2.0 data.

Summary by alphaXiv.