Reproduce “Hybrid Hypergraph Networks for Multimodal Sequence Data Classification” on OpenResearch

About this paper

Feng Xu, Hui Wang, Yuting Huang, Danwei Zhang, Zizhu Fan

This paper introduces Hybrid Hypergraph Networks (HHN) for multimodal sequence data classification, which models high-order intra-modal dependencies using entropy-guided hypergraphs and robust inter-modal relationships through temporal-weighted Graph Attention Networks. The framework achieves state-of-the-art performance across diverse benchmarks, including AudioSet and FMF industrial fault diagnosis, using a compact model with 2.1 million parameters.

Key insights

• Hypergraphs are crucial for capturing high-order relationships and structural heterogeneity within individual modalities, providing superior intra-modal representation compared to traditional graphs.
• An entropy-guided adaptive window mechanism for hyperedge construction effectively identifies and connects structurally diverse nodes, enhancing the discriminative power of the intra-modal representation.
• Robust multimodal classification necessitates both effective intra-modal processing (via hypergraphs) and temporally aligned inter-modal fusion (via GATs and Hawkes process), with their synergy being critical for performance.

Summary by alphaXiv.