Reproduce “EditDuet: A Multi-Agent System for Video Non-Linear Editing” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Marcelo Sandoval-Castaneda, Bryan Russell, Josef Sivic, Gregory Shakhnarovich, Fabian Caba Heilbron

EditDuet, a multi-agent system, automates non-linear video editing from natural language instructions by employing an Editor agent and a Critic agent, enhanced through self-supervised exploration for inter-agent communication. The system generated B-roll sequences with 86.9% human preference and achieved an 8.2% failure rate on the EditStock dataset, demonstrating improved quality and robustness over existing methods.

Key insights

  • Formulating non-linear video editing as an iterative, multi-agent decision-making process using specialized LLM agents significantly advances the automation of complex creative tasks.
  • Self-supervised in-context learning is crucial for teaching agents effective communication, drastically reducing system failures and improving output quality in collaborative LLM-based systems.
  • Vision-Language Models can reliably function as automated judges for subjective creative tasks, offering a scalable and cost-effective alternative to human evaluation.
  • LLMs can orchestrate complex visual manipulation tasks by interpreting structured visual metadata and executing predefined NLE functions, extending their utility beyond text-centric domains.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.