Reproduce “I'm Fine, But My Voice Isn't: Cross-Modal Affective Dissonance Detection for Reflective Journaling” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Sumin Lee
Researchers introduced Cross-Modal Affective Dissonance Detection (CADD) to identify discrepancies between expressed text and vocal emotions, developing a Dissonance-Aware Cross-Modal (DACM) model that achieved a 0.711 F1-score on a synthetic dataset. The work also quantified a substantial domain gap, finding that the model's performance dropped to near random chance (e.g., 0.362 F1 on IEMOCAP) when applied to naturalistic speech.
Key insights
- Cross-modal divergence between text and acoustic emotional expression can be a meaningful signal for detecting self-concealment or emotion regulation strategies, rather than simply being noise.
- Asymmetric cross-modal attention is a crucial architectural component for effective multimodal fusion, resolving gradient degeneracy issues inherent in standard pooled fusion methods and providing semantically contextualized features.
- A significant domain gap exists between TTS-synthesized affective dissonance and naturalistic human speech, implying that models trained on controlled synthetic data may not generalize well to real-world spontaneous expressions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.