Reproduce “Agentic Coding in the Wild: Characterizing GitHub Copilot Traces at Production Scale” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Banruo Liu, Haoran Qiu, Íñigo Goiri, Rodrigo Fonseca, Ricardo Bianchini, Esha Choukse
The authors analyze sampled Copilot coding-agent telemetry from 13.5 million sessions involving 3.2 million users. They find that 87% of LLM calls are agent-initiated and measure cache-hit rates across within-turn calls and turn boundaries, including model switches. An offline predictor reaches ROC-AUC 0.73 for gaps over 60 seconds, but no deployed resource savings are measured.
Key insights
- AI coding agents exhibit tight LLM↔tool coupling (near 1:1 call ratio) and high autonomy, with approximately 87% of LLM calls being agent-initiated, leading to complex, stateful workflows.
- KV cache dynamics are critical: while intra-turn hit rates are high (~92-94%), they degrade significantly at turn boundaries (26% drop) and with context compaction (66% drop), challenging current cache management strategies.
- User behavior and workflow archetypes are highly heterogeneous; a small fraction of intensive users (Coders, Deep-loop) drives the bulk of serving load, with a 50x difference in per-turn token consumption between archetypes.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.