Reproduce “Disjoint Mapping Network for Cross-modal Matching of Voices and Faces” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yandong Wen, Mahmoud Al Ismail, Weiyang Liu, Bhiksha Raj, Rita Singh
The Disjoint Mapping Network (DIMNet) framework from Carnegie Mellon University and Georgia Institute of Technology facilitates cross-modal matching of voices and faces by training separate modality embeddings to predict common covariates like identity and gender. This approach achieved 83.45% accuracy in voice-to-face 1:2 matching, outperforming prior state-of-the-art, and yielded an approximately 25% Equal Error Rate for cross-modal verification.
Key insights
- Learning common representations through explicit supervision from shared covariates is an effective and robust alternative to direct inter-modal mapping for cross-modal association.
- A substantial portion of previous direct mapping performance can be attributed to implicitly learning covariate information, which DIMNet explicitly disentangles.
- Identity (ID) proves to be the most informative covariate for cross-modal matching, with gender providing marginal, complementary benefits when combined with ID supervision.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.