Reproduce “MoFu: Scale-Aware Modulation and Fourier Fusion for Multi-Subject Video Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Run Ling, Ke Cao, Jian Lu, Ao Ma, Haowei Liu, Runze He, Changwei Wang, Rongtao Xu and 10 more
MoFu is a framework for multi-subject video generation that resolves scale inconsistencies and permutation sensitivity through a Diffusion Transformer backbone. It employs Scale-Aware Modulation, guided by textual prompts, and Fourier Fusion for permutation-invariant reference image aggregation. MoFu achieved top performance on its new MoFu-Bench benchmark, scoring highest in Aesthetics (0.401), FaceSim (0.396), GmeScore (0.745), ScaleScore (0.585), and SubjectSim (0.755).
Key insights
- Explicitly capturing and applying scale relationships derived from textual prompts through adaptive modulation is effective for generating subjects at natural and consistent sizes.
- Aggregating visual features from multiple reference images in the frequency domain, rather than the spatial domain, enables a robust and permutation-invariant fusion process.
- A combined loss function that simultaneously enforces scale consistency and permutation invariance during training is crucial for developing reliable and robust multi-subject video generation models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.