•  
  •  
 

Corresponding Author ORCID Identifer

Oluwasegun Abiodun Abioye https://orcid.org/0009-0001-4100-9736

Section

Regular Issue

Abstract

The growing sophistication of phishing attacks requires better detection than classic conventional filters. Previous work has focused on the accuracy, precision and recall of classifiers but there is a lack in testing well-calibrated probability predictions, which are crucial for reliable decision-making using cybersecurity systems. To fill this gap, we present in this work a phishing email detection framework that combines semantically rich Sentence-BERT all-mpnet-base-v2 model embeddings with a wide variety of machine learning classifiers. A total of nine model configurations were tested, including Sentence-BERT combined with GRU, SVM, Random Forest, Logistic Regression, k-Nearest Neighbours, Gaussian Naïve Bayes, Multi-Layer Perceptron, LightGBM (LGBM), and Extra Trees. Experiments were performed on two published datasets, evaluated in terms of most common classification measures used in practice: accuracy, precision, recall and F1-score as well as Brier score for probability calibration. Results indicate that the SVM and MLP models obtain state-of-the-art accuracies of up to 0.9869 and 0.9853, respectively, alongside excellent calibration with the lowest Brier scores. Although the GRU also achieves well-calibrated probabilities, its accuracy exhibits significant variability across datasets, dropping to 0.9204 on the second dataset. Meanwhile, LGBM is shown to be an efficient candidate with strong performance and reliability, whereas tree-based ensembles achieve high prediction performance but require post-hoc calibration. The results reveal the importance of obtaining well-calibrated probabilities in order to facilitate the actual use of phishing detection models within real-world email security operations.

Download

Share

COinS