Reproduce “Complex Agents, Shallow Tests: Demystifying and Enhancing Test Adequacy of Agent Harness in the Wild” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yifan Xiong, Jingyi Ge, Zhenpeng Chen, Yiling Lou
HarnessTester improves testing of LLM-dependent harness (LDH) code, the agent software whose behavior depends on model outputs. It guides coverage-driven test generation with agent-harness contracts that preserve expected data shapes and interactions, plus repository context retrieval. Across ten projects, after 120 minutes per project using GPT-5-mini, macro-average LDH coverage gains exceeded scoped baselines; results concern targeted files, not whole projects.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.