Reproduce “Quantifying and Enhancing Multi-modal Robustness with Modality Preference” on OpenResearch

About this paper

Zequn Yang, Yake Wei, Ce Liang, Di Hu

A theoretical framework from Renmin University of China quantifies multi-modal robustness, revealing how modality preference creates uneven vulnerabilities in AI models. This understanding informs their Certifiable Robust Multi-modal Training (CRMT) method, which improves resilience against diverse attacks and missing modalities, reducing performance disparities across different input types.

Key insights

• Multi-modal robustness is mathematically quantifiable, dependent on the individual discriminative strength (uni-modal representation margins) and the reliability of how modalities are integrated.
• Modality preference causes an "imbalance problem," where models rely too heavily on a single modality, resulting in fragile representations for others and amplified vulnerability to attacks on the preferred input.
• The proposed orthogonal-based framework simplifies the optimization of multi-modal robustness by decoupling representation learning from modality integration, enabling direct maximization of a certified lower bound.

Summary by alphaXiv.