Reproduce “Generating Expressive and Customizable Evals for Timeseries Data Analysis Agents with AgentFuel” on OpenResearch

About this paper

Aadyaa Maddi, Prakhar Naval, Deepti Mande, Shane Duan, Muckai Girish, Vyas Sekar

Rockfish Data and Carnegie Mellon University researchers introduce AgentFuel, a framework for generating expressive and customizable evaluations tailored for timeseries data analysis agents. The framework reveals that agents achieve an average accuracy of 66% on e-commerce and 60% on IoT benchmarks, but only 21% on a telecom benchmark, particularly struggling with stateful (34% accuracy) and incident-specific queries (10% accuracy), and demonstrates that integrating AgentFuel into an optimization loop can improve agent accuracy by up to 25%.

Key insights

• Existing data analysis agents exhibit a significant performance degradation when faced with stateful and incident-specific queries over timeseries data, achieving considerably lower accuracy compared to stateless queries.
• Agent performance varies notably across different agent architectures and underlying LLMs, sometimes showing that 'more capable' LLMs do not inherently outperform 'simpler' ones on these specialized benchmarks.
• Primary failure modes include schema confusion, inability to track state across events within a session (e.g., sequence understanding), and a lack of data exploration capabilities for dynamic incident detection rather than assuming fixed time windows.

Summary by alphaXiv.