Reproduce “Learning to Compose Neural Networks for Question Answering” on OpenResearch

About this paper

Jacob Andreas, Marcus Rohrbach, Trevor Darrell, Dan Klein

Dynamic Neural Module Networks (D-NMNs) are introduced to learn compositional neural network structures for question answering by dynamically assembling neural modules for specific questions. This approach achieves state-of-the-art accuracy on Visual Question Answering (VQA) and geographical question answering (GeoQA) datasets, demonstrating effective reasoning across both continuous image data and structured knowledge bases.

Key insights

• Combining explicit linguistic compositionality with continuous and differentiable neural representations creates a flexible and powerful reasoning framework, leveraging strengths from both symbolic and connectionist AI paradigms.
• Dynamically tailoring the network structure to each question, instead of using a fixed architecture, allows for more adaptable, efficient, and potentially interpretable computation.
• Reinforcement learning enables the joint optimization of both module behaviors and the dynamic network assembly process using only indirect supervision, significantly reducing annotation requirements.

Summary by alphaXiv.