Comparison of the Effectiveness of Machine Learning Models in the Area of Security Breach Detection

As the number of new and increasingly difficult to detect threats increases, the effectiveness of classification methods in intrusion detection systems (IDS) becomes particularly important. Previous research approaches mainly focus on detecting incidents in a binary approach, while issues related to multi-class classification, which includes recognizing multiple types of attacks simultaneously, remain insufficiently explored. This paper attempts to fill this gap by comparing the effectiveness of four popular machine learning algorithms – Support Vector Machine (SVM), Naive Bayes (Gaussian and Bernoulli variants), Random Forest, and XGBoost – evaluating their performance in both binary and multiclass classification tasks on the real CIC-IDS-2017 dataset. The data was subjected to comprehensive preprocessing, including cleaning, dimensionality reduction, and class balancing. Experiments assessed models based on accuracy, precision, recall, F1-score (macro and weighted averages), and computational efficiency. Results demonstrate that ensemble methods, particularly Random Forest and XGBoost, significantly outperform others in detection accuracy, with F1-scores exceeding 98% in binary and 85% (macro average) in multiclass classification after hyperparameter tuning. SVM showed solid performance but with higher computational costs, while naive Bayes models offered fast training but lower detection effectiveness. The findings confirm the suitability of tree-based ensemble models for intrusion detection systems (IDS), highlighting their robustness, scalability, and accuracy in identifying both known and novel threats.