Reproduce “OmniInteract: Benchmarking Real-World Streaming Interaction for Real-Time Omnimodal Assistants” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Xudong Lu, Xueying Li, Annan Wang, Yang Bo, Jinpeng Chen, Zengliang Li, Nianzu Yang, Rui Liu and 3 more

OmniInteract benchmarks real-time audio-visual assistants by scoring spoken requests and visual events in time-bounded response slots. Its IA-QTF1 metric combines answer quality and timing with penalties for missed or inappropriate responses. Across four evaluated models, MiniCPM-o 4.5 has the highest all-slots score, 0.368; AURA has the highest long-horizon monitoring score, 0.052, on this benchmark.

Key insights

  • Current omnimodal real-time interaction models demonstrate significant limitations in robust streaming interaction, particularly concerning long-horizon task monitoring and maintaining context across interruptions.
  • The cognitive load associated with full-duplex online interaction (simultaneous listening, visual processing, and response generation) leads to a substantial degradation in core reasoning quality compared to offline observation.
  • Models frequently treat nested queries as permanent context switches, often failing to resume the outer query after addressing an inserted inner query.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.