Reproduce “DoraemonGPT: Toward Understanding Dynamic Scenes with Large Language Models (Exemplified as A Video Agent)” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zongxin Yang, Guikun Chen, Xiaodi Li, Wenguan Wang, Yi Yang
DoraemonGPT presents an LLM-driven agent framework for comprehensive dynamic scene understanding in videos, leveraging a structured memory, diverse tools, and a Monte Carlo Tree Search planner. This system outperforms prior LLM-driven methods and even supervised models on video question answering and referring object segmentation benchmarks, demonstrating robust spatial-temporal reasoning and knowledge integration.
Key insights
- Decomposing video information into task-relevant, structured symbolic memory (TSM) drastically improves LLM efficiency and accuracy for spatial-temporal reasoning by reducing context and enabling precise querying.
- Employing a Monte Carlo Tree Search (MCTS) planner for action sequencing allows thorough exploration of complex planning spaces, leading to more robust and higher-quality solutions compared to greedy approaches.
- Modular knowledge tools provide crucial extensibility, enabling LLM agents to overcome their inherent knowledge limitations and address domain-specific tasks by seamlessly integrating external information sources.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.