Reproduce “JevAgentHarness: Benchmarking, Deploying and Training Decision Models That Share Control with LLMs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Kunlun Zhu, Tao Feng, Xu Han, Heng Wang, Ling Yang, Xiangru Tang, Jiaxuan You, Heng Ji
JevAgentHarness evaluates small decision models scoring listed actions, with a confidence threshold sending uncertain choices to a large language model (LLM). The authors test running agents and train on states they reach. On shifted DecisionGym tasks, AgentDecision-4B achieved 91.5% task success versus Jev’s 82.0% when making every tool choice. New tasks may require refitting.
Key insights
- Standalone accuracy is an unreliable proxy for agent success; step-level evaluation is more informative, while live execution captures downstream effects that recorded-state tests cannot.
- Confidence-based delegation can reduce costs while limiting risk, but fixed thresholds or unrestricted delegation can sharply reduce task success.
- Training and calibration are specific to an agent’s rules and state distribution; distribution shifts can still produce confident, costly errors, especially for irreversible actions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.