Reproduce “MSR: Multiple Subject Reference for Video Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Guannan Li, Jiaji Chen, Jingyuan Liao, Yu Geng, Baolan Qiu
Multiple Subject Reference (MSR) conditions video generation on separate reference images by independently encoding each image, adding a slot embedding and positional shift, and prepending the clean reference tokens to noisy target-video tokens. Text assigns each source a role. Clips show character cues and environments, while qualitative comparisons suggest fewer clothing confusions; no score establishes improvement or identity separation.
Key insights
- A slot marks the input source, not the image’s semantic category or the identity of a generated subject; language remains responsible for specifying how each reference should be used.
- Separately encoding references and providing explicit source cues avoids representing unrelated images as one continuous clip, but neither cue alone guarantees correct subject binding or consistency over motion.
- Reference fidelity is uneven: simple clothing, objects, and scenes were easier to maintain than similar-looking characters, complex garments, or subjects that turn or reappear after a cut.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.