Reproduce “AgentLens: Production-Assessed Trajectory Reviews for Coding Agent Evaluation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Andrey Podivilov, Vadim Lomshakov, Sergey Savin, Matvei Startsev, Roman Pozharskiy, Maksim Parshin, Sergey Nikolenko
AgentLens evaluates coding agents by combining repository checks with reviews of complete interaction trajectories. Its reviews assess outcomes, instruction following and tool use, citing evidence from sessions. In a production regression, a side-by-side review identified a ConcurrentModificationException during parallel tool calling. This illustrates diagnostic potential, not validated accuracy; the released evaluation is a compact Java-only fold, and judge-human agreement remains preliminary.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.