Abstract: The intrusion detection scheme based on system call in the traditional host domain often monitors the running behavior of a single privileged process. It is difficult to effectively detect the abnormal process behavior of the virtual machine using the host intrusion detection scheme because of more security risks in the cloud computing environment. To break this limitation, a virtual machine process behavior detection model based on system call vector space is proposed. The model collects system call data of different operating system without using agent in the virtual machine. The TF-IDF (term frequency-inverse document frequency) algorithm idea is introduced to weight the process system call data to distinguish different running services in the virtual machine and identify abnormal process behavior. Furthermore, in order to optimize the efficiency of the detection algorithm, a storage strategy combining compressed sparse row (CSR) matrix and K-dimension tree is designed. Eventually a prototype system called VMPBD (virtual machine process behavior detecting) has been implemented on the platform of KVM (kernel-based virtual machine). The functions and performance of VMPBD is tested on Linux and Windows virtual machines. The results show that VMPBD can effectively detect the abnormal behavior of the virtual machine processes, and the detection false alarm rate and system performance overhead are within the acceptable range.

Key words: virtualization, anomaly detection, system call analysis, vector space, kernel-based virtual machine