Reproduce “Sherlock: Reliable and Efficient Agentic Workflow Execution” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yeonju Ro, Haoran Qiu, Íñigo Goiri, Rodrigo Fonseca, Ricardo Bianchini, Aditya Akella, Zhangyang Wang, Mattan Erez and 1 more
Sherlock is a system designed to improve the reliability and efficiency of agentic Large Language Model (LLM) workflows. It achieves an 18.3% average accuracy gain and reduces workflow execution time by up to 48.7% over non-speculative baselines, while lowering verification costs by 26.0% through intelligent verifier placement, dynamic selection, and speculative execution.
Key insights
- Workflow nodes exhibit varying degrees of vulnerability; targeting verification resources to terminal and initial nodes, and those with high fan-in, significantly improves overall accuracy per unit of cost.
- The efficacy and cost of different LLM verifiers are highly task-dependent, necessitating a dynamic and learning-based approach for selecting the optimal verifier to achieve Pareto-optimal accuracy-cost trade-offs.
- Speculative execution, combined with selective rollback based on lightweight semantic similarity checks, can effectively hide verification latency, making reliable agentic workflows suitable for online applications.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.