vDefender: An explainable and introspection-based approach for identifying emerging malware behaviour at hypervisor-layer in virtualization environment

Virtualization can be defined as the backbone of cloud computing services, which has gathered significant attention from organizations and users. Due to the increasing number of cyberattacks, virtualization security has become a crucial area of study. In this paper, we propose an explainable and introspection-based malware detection approach called vDefender for fine-grain monitoring of virtual machine (VM) processes at the hypervisor to identify the malicious behaviour of 17 different malware families of Windows exhibiting new evolving behaviour. Initially, it performs a basic security check to detect hidden processes and ensures the presence of security-critical processes. Then, deep memory introspection is performed using a software breakpoints injection approach to intercept the execution of processes. Various process activity logs are captured that include process-related, file manipulation, kernel heap object creation, exception-related activities, etc. Hybrid feature vectors are derived from these logs, which are reconstructed using the proposed mechanism to eliminate the redundant behaviour. The features are then learnt using Random Forest (RF) algorithm to classify distinct malware families. The interpretation and analysis of RF results involve the use of explainability techniques. The proposed approach achieves an accuracy of 95.49%, F1-score of 95.82% with 0.05% false alarms when evaluated using an emerging malware dataset. The contribution includes a comprehensive discussion of results, accompanied by a comparative analysis of current approaches that gives readers insight towards future research directions.