Reproduce “Understanding the AI-powered Binary Code Similarity Detection” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Lirong Fu, Peiyu Liu, Wenlong Meng, Kangjie Lu, Shize Zhou, Xuhong Zhang, Wenzhi Chen, Shouling Ji

This research systematically evaluates 15 state-of-the-art AI-powered Binary Code Similarity Detection approaches, comparing their performance across different architectures and optimization levels, and analyzing their effectiveness in vulnerability search and license violation detection. The study identified "embedding collision" as a significant GNN limitation causing 95% of false positives and demonstrated that common metrics like AUC do not fully reflect practical search performance.

Key insights

  • GNN-based BinSD approaches excel in similar function detection but face substantial performance degradation in cross-architecture scenarios.
  • "Embedding collision," rather than over-smoothing, is a primary limitation for GNNs in BinSD, causing 95% of false positives in function search.
  • Traditional classification metrics like AUC often do not accurately reflect practical retrieval performance, where ranking metrics are more indicative.
  • The effectiveness of BinSD tools varies across downstream applications, performing better in license violation detection than in vulnerability search, which still requires significant manual effort.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.