Reproduce “GEMMAS: Graph-based Evaluation Metrics for Multi Agent Systems” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jisoo Lee, Raeyoung Chang, Dongwook Kwon, Harmanpreet Singh, Nikhil Verma
The paper introduces GEMMAS, a graph-based evaluation framework designed to quantify collaboration quality and resource utilization within multi-agent LLM systems. It defines Information Diversity Score (IDS) and Unnecessary Path Ratio (UPR) to diagnose internal inefficiencies, revealing significant differences in collaboration patterns and redundancy even among systems with similar final task accuracies, such as G-Designer achieving a fivefold reduction in UPR compared to a baseline on GSM8K.
Key insights
- GEMMAS effectively exposes substantial internal inefficiencies, such as high redundancy, that are completely masked by traditional outcome-only metrics like final answer accuracy.
- Systems achieving identical task accuracy can exhibit significantly different internal collaboration patterns, efficiency, and semantic diversity, indicating that final accuracy is an insufficient measure of system quality.
- The IDS and UPR metrics provide clear, complementary signals that guide MAS designers in identifying optimal configurations that balance diverse, unique contributions with efficient, non-redundant communication.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.