Reproduce “CLOSURE: Assessing Systematic Generalization of CLEVR Models” on OpenResearch

About this paper

Dzmitry Bahdanau, Harm de Vries, Timothy J. O'Donnell, Shikhar Murty, Philippe Beaudoin, Yoshua Bengio, Aaron Courville

Researchers created CLOSURE, a benchmark designed to evaluate systematic generalization in Visual Question Answering (VQA) models by presenting novel combinations of familiar linguistic constructs. The study found that while most state-of-the-art VQA models struggled with this out-of-distribution challenge, a new Vector-NMN architecture, using ground-truth programs and vector-valued module interfaces, largely maintained its performance on CLOSURE, nearing its original CLEVR accuracy.

Key insights

• High accuracy on the CLEVR dataset does not reliably indicate systematic generalization, as most evaluated models experienced substantial performance drops on the CLOSURE benchmark.
• The design of inter-module interfaces in modular networks critically impacts systematic generalization; constrained, vector-valued interfaces in Vector-NMN promoted better generalization compared to high-capacity tensor interfaces.
• Program-based models, under strong supervision, exhibited a near-complete adaptation to novel compositional tasks with few-shot learning, contrasting with the more partial adaptation observed in end-to-end models.

Summary by alphaXiv.