Reproduce “DIME: Fine-grained Interpretations of Multimodal Models via Disentangled Local Explanations” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yiwei Lyu, Paul Pu Liang, Zihao Deng, Ruslan Salakhutdinov, Louis-Philippe Morency
A method called DIME (Disentangled Multimodal Explanations) is introduced to interpret black-box multimodal models by decomposing their output into disentangled unimodal contributions and multimodal interactions. This approach generates fine-grained, localized explanations for each component using LIME, achieving up to 0.98 Pearson correlation with ground truth on synthetic data and significantly improving human confidence and inter-annotator agreement in identifying dominant factors in VQA.
Key insights
- Decomposing a multimodal model's output into distinct unimodal contributions and multimodal interactions offers a significantly more granular and reliable understanding of model predictions.
- Applying local explanation techniques like LIME to these disentangled components effectively generates fine-grained, modality-specific visualizations of both individual and interactive influences.
- DIME's disentanglement method reliably separates these components, confirmed by the stability of unimodal explanations against changes in other modalities and the sensitivity of interaction explanations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.