Reproduce “DuoFlow: Towards Synchronized Speech and Motion Generation with Fine-Grained Control” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Jun Dong, Ronghui Li, Zhongyuan Hu, Yunqi Zhao, Lijie Guo, Ruizhang Xu, Di Kang, Linchao Bao and 1 more

DuoFlow jointly generates speech and full-body motion from a transcript, a voice description, and either a coarse or detailed motion instruction. It encodes the two streams separately and denoises them together, with bidirectional interaction. On MotionSpeech, its L1/L2 Global PairAcc is 87.43%/90.40%, an automatic compatibility measure, not human judgment or direct evidence of instruction following.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.