Reproduce “Evaluation of an Audio-Video Multimodal Deepfake Dataset using Unimodal and Multimodal Detectors” on OpenResearch

About this paper

Hasam Khalid, Minha Kim, Shahroz Tariq, Simon S. Woo

Researchers from Sungkyunkwan University systematically evaluated the FakeAVCeleb multimodal deepfake dataset using unimodal, ensemble, and adapted general multimodal detectors. The study revealed that existing detection paradigms show limited efficacy against complex audio-video deepfakes with lip-synced synthesized audio, with the best ensemble method reaching 78.04% accuracy and adapted multimodal methods performing poorly.

Key insights

• State-of-the-art unimodal deepfake detectors exhibit inconsistent performance on the FakeAVCeleb dataset, often struggling with one modality while performing moderately on the other.
• Ensemble methods, while providing some minor improvements over individual unimodal baselines, are insufficient for achieving high detection accuracy against complex audio-video deepfakes.
• General-purpose multimodal learning models, when adapted for deepfake detection, perform poorly, demonstrating their inability to effectively generalize to the specific challenges of audio-visual deepfake detection.

Summary by alphaXiv.