Reproduce “Distilling Cross-Modal Knowledge via Feature Disentanglement” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Junhong Liu, Yuan Zhang, Tao Huang, Wenchao Xu, Renyu Yang

Researchers from Beihang, Peking, Shanghai Jiao Tong, and HKUST developed Frequency-Decoupled Cross-Modal Knowledge Distillation (FD-CMKD), a framework that disentangles cross-modal features based on frequency to address representational conflicts. The method consistently achieves state-of-the-art performance across multimodal classification and semantic segmentation tasks, improving, for example, AVE visual modality performance by 9% and reaching 33.2% mIoU for depth on NYU-Depth V2.

Key insights

  • Low-frequency feature components predominantly encode modality-generic semantic information, showing strong cross-modal consistency.
  • High-frequency feature components capture modality-specific details or noise, which require more relaxed distillation due to their inherent heterogeneity.
  • Inter-modal feature scale and distribution differences are a fundamental challenge in cross-modal knowledge transfer, requiring explicit alignment mechanisms.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.