Reproduce “LM Agents for Coordinating Multi-User Information Gathering” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Harsh Jhamtani, Jacob Andreas, Benjamin Van Durme
IBM Research introduced PEOPLEJOIN, a benchmark for evaluating large language model agents in coordinating multi-user information gathering and document creation tasks within simulated organizational settings. Experiments with state-of-the-art LLMs, including GPT-4-turbo, yielded a maximum `Match` score of 54.8% on question-answering tasks, indicating the challenging nature of active, conversational information acquisition.
Key insights
- Current state-of-the-art LLM agents exhibit limitations in strategic information acquisition, precise query formulation, and robust task orchestration within dynamic multi-user environments.
- Explicit self-reflection through a 'Reactive' agent architecture generally improves performance on complex, multi-hop question answering tasks, but its impact is less pronounced for open-ended document creation.
- The LLM-based user simulators within PEOPLEJOIN provide interactions that are qualitatively similar to human collaborators, validating the benchmark's realism, though human nuance can lead to slightly different outcomes.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.