SECURITY AND COMMUNICATION NETWORKS, 2018 (SCI-Expanded)
In computer security, masquerade detection is a special type of intrusion detection problem. Effective and early intrusion detection is a crucial factor for computer security. Although considerable work has been focused on masquerade detection for more than a decade, achieving a high level of accuracy and a comparatively low false alarm rate is still a big challenge. In this paper, we present a comprehensive empirical study in the area of anomaly-based masquerade detection using three deep learning models, namely, Deep Neural Networks (DNN), Long Short-Term Memory Recurrent Neural Networks (LSTM-RNN), and Convolutional Neural Networks (CNN). In order to surpass previous studies on this subject, we used three UNIX command line-based datasets, with six variant data configurations implemented from them. Furthermore, static and dynamic masquerade detection approaches were utilized in this study. In a static approach, DNN and LSTM-RNN models are used along with a Particle Swarm Optimization-based algorithm for their hyperparameters selection. On the other hand, a CNN model is employed in a dynamic approach. Moreover, twelve well-known evaluation metrics are used to assess model performance in each of the data configurations. Finally, intensive quantitative and ROC curves analyses of results are provided at the end of this paper. The results not only show that deep learning models outperform all traditional machine learning methods in the literature but also prove their ability to enhance masquerade detection on the used datasets significantly.