A Polymorphic Shellcode Detection Method Based on Dual-Mode Virtual Machine

Malicious code such as virus and worm propagates and attacks via the Internet which compromises network securities seriously. By exploiting vulnerabilities of services running on a remote host, the malicious code can inject shellcode into the host and gain complete control over it. Recently, shellcode attacks tend to evade network detection by employing polymorphic techniques. However, previous detection methods lack the real-time performance, resilience against evasions and the consideration of distinguishing polymorphic shellcode from normal shell protected code. We propose an enhanced polymorphic shellcode detection approach based on dual-mode virtual machine. We firstly filter network traffic via an improved GetPC location mechanism and IA-32 instruction recognition method, subsequently implement a dual-mode virtual machine to execute the target instruction flow by transferring states of the finite automaton between control-flow mode and data-flow mode according to various decision conditions. A prototype system has been implemented for our approach and experimental results indicate that when detecting real network data mixed with the polymorphic shellcode samples, the approach we proposed succeeds in differentiating the polymorphic shellcode from shell protected software without losing its accuracy and its time complexity lies between the static analysis and dynamic emulation, which makes it an encouraging method for network attack detection.