Reproduce “OfficeBench: Benchmarking Language Agents across Multiple Applications for Office Automation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zilong Wang, Yuedong Cui, Li Zhong, Zimin Zhang, Da Yin, Bill Yuchen Lin, Jingbo Shang
OFFICEBENCH introduces a benchmark for evaluating large language model agents in complex, multi-application office automation workflows, assessing their ability to plan and execute tasks across applications like Word, Excel, and Email. Evaluations reveal GPT-4 Omni achieved a 47% pass rate, while Llama 3 (70B-Instruct) reached 27.33%, demonstrating a substantial gap compared to human performance of 93.33% and identifying challenges in cross-application planning.
Key insights
- A substantial gap exists between the performance of state-of-the-art LLM agents (e.g., GPT-4 Omni at 47%) and human efficiency (93.33%) in office automation tasks.
- Agent performance degrades considerably as the number of applications required for a task increases, highlighting challenges in multi-application coordination and planning.
- The dedicated `switch_app` mechanism enhances performance and reduces token usage, suggesting structured action spaces are beneficial for LLM agents.
- Common failure modes include getting stuck in redundant operations, hallucinating non-existent actions, and struggling with complex, multi-step planning across different applications.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.