Reproduce “ViDove: A Translation Agent System with Multimodal Context and Memory-Augmented Reasoning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yichen Lu, Wei Dai, Jiaen Liu, Ching Wing Kwok, Zongheng Wu, Xudong Xiao, Ao Sun, Sheng Fu and 4 more
Pigeon AI and collaborators developed ViDove, a multi-agent translation system that integrates visual, audio, and textual inputs for long-form video content translation, outperforming baselines by 28.8% in BLEU score on the newly introduced DoveBench dataset. The system simulates human translator workflows using specialized agents and memory-augmented reasoning.
Key insights
- Integrating specialized agents for auditory and visual perception, combined with a robust memory system, is highly effective for multimodal machine translation, especially for long-form content.
- Domain-specific memory and a dedicated proofreader agent are crucial components for maintaining terminology accuracy, grammatical correctness, and overall translation quality.
- Monolithic Multimodal Large Language Models (MLLMs) struggle with long-form video inputs and complex instruction following, highlighting the advantage of ViDove's modular, multi-agent architecture.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.