Reproduce “ProEvent: An Event-centric Benchmark for Proactive Agents” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Guanzhen Li, Liangming Pan, Leye Wang
Peking University researchers introduced PROEVENT, an event-centric benchmark designed to evaluate proactive agents' ability to autonomously maintain a user's timetable from instant messaging chats. Evaluations using this benchmark reveal that current large language models (LLMs) frequently overreact and struggle with implicitly conveyed event changes and complex chat dynamics, with the best model achieving a Timetable Success Rate of 27.2%.
Key insights
- Current LLMs exhibit a consistent tendency towards overreaction (high False Detection Rate), initiating proactive actions even when not required, though deliberative reasoning pipelines can mitigate this.
- Detecting implicit event cancellations and managing the complexities of real-world chat environments (e.g., negotiation turns, concurrent threads, semantic noise) pose significant challenges for LLM-based agents.
- LLMs demonstrate fundamental deficiencies in crucial reasoning abilities, including extracting implicit event information from narrative dialogue, performing temporal calculations, and adopting a first-person user perspective.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.