Reproduce “OmniVChat: Synthesizing, Benchmarking, and Training for Native Audio-Visual Dialogue” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Haolin He, Yunfei Chu, Qi Chen, Wen Huang, Yuan Feng, Muzhi Zhu, Zheqi Dai, Haoning Xu and 10 more
OmniVChat trains an omni model to connect spoken questions with visual evidence and answer in text, without a separate typed question or transcript. OmniVChat-Studio synthesizes dialogues and rubrics, while rubric, efficiency, and style rewards guide training. Qwen3-Omni-Instruct improved Mean from 0.465 to 0.652 and Human from 0.402 to 0.632 on a limited Chinese single-turn probe.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.