Reproduce “MuKEA: Multimodal Knowledge Extraction and Accumulation for Knowledge-based Visual Question Answering” on OpenResearch

About this paper

Yang Ding, Jing Yu, Bang Liu, Yue Hu, Mingxin Cui, Qi Wu

The MuKEA framework advances knowledge-based Visual Question Answering (KB-VQA) by explicitly extracting and accumulating multimodal knowledge triplets (visual object, implicit relation, fact answer) directly from VQA samples. This system achieved 42.59% accuracy on the OK-VQA dataset, surpassing previous state-of-the-art by 3.35%, and demonstrated strong generalization to long-tail answers.

Key insights

• Explicitly representing complex, vision-relevant facts as multimodal triplets provides explainable reasoning and captures high-order relationships that are inexpressible in traditional text-based KBs.
• The multimodal knowledge extracted by MuKEA is complementary to existing text-based knowledge graphs, suggesting that future KB-VQA systems could benefit from hybrid architectures.
• An end-to-end learning approach for knowledge extraction and accumulation mitigates cascading errors inherent in multi-stage retrieval-and-read pipelines, leading to more robust performance.

Summary by alphaXiv.