Reproduce “The Matthew Effect of AI Programming Assistants: A Hidden Bias in Software Evolution” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Fei Gu, Zi Liang, Jiahao MA, Hongzong LI

Researchers empirically demonstrated a "Matthew effect" in AI programming assistants, where models systematically favor popular languages and frameworks, resulting in significantly higher code generation success rates for mainstream technologies and a quantifiable "AI productivity tax" for less prevalent ones. This bias, consistently observed across multiple state-of-the-art LLMs, impacts long-term software ecosystem diversity.

Key insights

  • AI programming assistants exhibit a strong "Matthew effect," systematically amplifying existing popularity hierarchies among programming languages and frameworks.
  • The performance gap between AI support for popular versus niche languages/frameworks widens with increasing problem difficulty and complexity.
  • Failures in popular languages often involve semantic errors, while niche languages frequently result in fundamental compile errors due to insufficient training data.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.