Reproduce “CodeSearchNet Challenge: Evaluating the State of Semantic Code Search” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, Marc Brockschmidt
CodeSearchNet pairs millions of documented functions from six programming languages with a human-judged benchmark of developer-style search queries, providing both a training resource and an evaluation framework for semantic code search. On the realistic challenge set, a simple neural bag-of-words model outperformed more complex encoders, reaching average NDCG scores of 0.574 on judged candidates and 0.340 across the full corpus, while self-attention performed best on the easier documentation-matching task with MRR 0.7011.
Key insights
- Performance on documentation-to-code matching does not predict performance on realistic developer queries: self-attention led the proxy task, whereas neural bag-of-words ranked first on the human-annotated challenge.
- Rare identifiers and exact lexical cues remain valuable in code search, helping explain why a simple bag-of-words model and ElasticSearch remained competitive against more contextual architectures.
- Relevance depends on factors beyond semantic similarity, including code quality, security, efficiency, project context, self-containment, ambiguity, and distinguishing a function from its inverse.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.