Reproduce “MG$^2$-RAG: Multi-Granularity Graph for Multimodal Retrieval-Augmented Generation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Sijun Dai, Qiang Huang, Xiaoxing You, Jun Yu

MG^2-RAG, a framework developed at Harbin Institute of Technology (Shenzhen), integrates a lightweight hierarchical multimodal knowledge graph construction with a multi-granularity graph retrieval mechanism. This approach reduces graph construction costs by an average of 23.9x and achieves state-of-the-art performance across multimodal retrieval, VQA, reasoning, and classification tasks, often surpassing human performance on specific benchmarks.

Key insights

  • A lightweight, rule-based approach to knowledge graph construction, combined with direct visual grounding of textual entities, significantly reduces the computational and financial overhead compared to MLLM-driven methods.
  • Fusing textual entities and visual objects into unified multimodal nodes and explicitly modeling their composition preserves fine-grained multimodal information, enabling more nuanced cross-modal understanding and reasoning.
  • A two-stage retrieval strategy that aggregates dense similarities and propagates relevance across a multi-granularity graph effectively uncovers structurally related evidence, overcoming limitations of flat vector retrieval for complex relational dependencies.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.