Interception and Identification of Guest OS Non-trapping System Call Instruction within VMM

To solve the problem that VMM can not monitor and control some Guest OS specific behavior due to its non-trapping feature in virtualized computing environment, an idea has been proposed to make those non-trapping instructions trap into VMM through modifying their normal execution conditions so as to cause system exception. According to the idea, special methods have been explored on how to intercept and identify the three different non-trapping system call instructions of x86 architecture from Guest OS within VMM. The int and sysenter instructions trap into VMM through causing GP system exception, while syscall instruction trap into VMM through causing UD system exception. They are identified with the virtual CPU context information within VMM. The Qemu&Kvm based prototype indicates that VMM can successfully intercept and identify all the three system call behaviors from Guest OS, and the performance overhead is within an accepted range for normal applications. For example, in unixbench shell test case, the performance overhead ratio is range 1900 to 2608. Compared with existing methods, they are all based on the architecture specification, so the advantage is that they are transparent to Guest OS and need not any modifications to Guest OS.