Reproduce “Decoupled Hierarchical Distillation for Multimodal Emotion Recognition” on OpenResearch

About this paper

Yong Li, Yuanzhi Wang, Yi Ding, Shiqing Zhang, Ke Lu, Cuntai Guan

Researchers developed Decoupled Hierarchical Multimodal Distillation (DHMD), a framework for multimodal emotion recognition that reduces cross-modal heterogeneity by decoupling features and employing a two-stage knowledge distillation strategy. This approach, which includes dynamic graph distillation and cross-modal dictionary matching, improved MER accuracy to 49.9% ACC7 on CMU-MOSI and 55.3% ACC7 on CMU-MOSEI, outperforming existing methods.

Key insights

• Decoupling multimodal features into homogeneous and heterogeneous spaces, combined with a self-regression mechanism and orthogonality constraints, effectively reduces cross-modal redundancy and enhances feature discriminability.
• A hierarchical knowledge distillation approach, combining coarse-grained dynamic graph distillation with fine-grained cross-modal dictionary matching, adaptively transfers knowledge across modalities and aligns semantics at different granularities.
• The proposed framework captures subtle inter-modal cues and improves robustness, as evidenced by its superior performance not only on general MER but also on specialized tasks like humor and sarcasm detection.

Summary by alphaXiv.