Reproduce “FUSED: Forensic-Semantic Mixture-of-Experts for AI Inpainting Detection and Localization” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Anton Nuzhdin, Marcel Worring, Ivona Najdenkoska
FUSED (Forensic-Semantic Mixture-of-Experts) is an AI framework developed at the University of Amsterdam for jointly detecting and localizing AI-generated inpainting manipulations in images. It integrates low-level forensic cues with high-level semantic features using a token-wise sparsely-gated Mixture-of-Experts architecture and a segmentation bridge, achieving state-of-the-art performance with a 44.5 average IoU for localization and an 88.0 average detection F1 across diverse generators on the OpenSDID benchmark.
Key insights
- Adaptive, token-level fusion of forensic and semantic features through a sparsely-gated MoE significantly improves generalization and robustness in inpainting detection and localization.
- Combining a forensic branch designed to suppress semantic content with a frozen foundation model's generalizable semantic understanding provides complementary strengths for robust detection.
- Directly supervising the fusion and routing modules with a pixel-level localization loss is crucial for achieving precise and consistent masks, especially under distribution shifts.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.