Reproduce “An Ensemble of Pre-trained Transformer Models For Imbalanced Multiclass Malware Classification” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Ferhat Demirkıran, Aykut Çayır, Uğur Ünal, Hasan Dağ

Researchers propose Random Transformer Forest (RTF), a bagging-based ensemble of pre-trained Transformer models, for imbalanced multiclass malware classification using API call sequences. The RTF model achieves state-of-the-art F1-scores on three diverse malware datasets, including a notable 0.6149 F1-score on the Catak dataset, and is the first to apply the tokenizer-free CANINE model in this domain.

Key insights

  • Transformer architectures, especially pre-trained models like BERT and CANINE, are more effective and efficient than LSTMs for API call sequence-based malware classification.
  • The tokenizer-free CANINE model is particularly well-suited for non-natural language sequences like API calls, proving effective for static API datasets.
  • Ensemble learning, specifically the proposed RTF combining pre-trained Transformers via bagging, significantly improves performance and robustness for imbalanced multiclass problems.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.