Abstract: A collaborative intrusion detection system (IDS) model, entitled virtual machine introspection & network-based IDS (VMI-N-IDS) is proposed, which is based on traditional introspection-based IDS and network-based IDS, for the defense of internal distributed denial of service (DDoS) attack threat of cloud cluster (e.g.cloud droplets freezing, CDF Attack). The CDF attack can exhaust the internal bandwidth of the cluster, the CPU and the memory resources of physical servers. Based on the game theory, IDS and attacker are treated as the two game parties in the VMI-N-IDS model. Utility functions of the two parties are supported, and it is proved that the game model is a non-cooperative and repeated game of incomplete information, and the subgame perfect Nash equilibrium is existent. Finally, the optimal defense strategy is proposed, which is the tradeoff between the false alarm rate and the malicious software size control, for solving the problem of dynamical adjustment strategy of internal intrude detection. The best strategy for the stages of IDS is to increase the threshold value β when the mathematical expectation of the suspicious value is greater than the load of server resources, and to reduce such value conversely. Experimental result shows that the proposed method can effectively defense the internal DDoS attack threat in the cloud environment.

Key words: cloud computing, network security, intrusion detection, DDoS attack, game theory