Abstract: Protocol anomaly detection, a new technique of anomaly detection, has great research value. Its incorporation with hidden Markov model (HMM) is still in infancy. In order to investigate the capabilities of hidden Markov model in this area, a protocol anomaly detection model based on HMM is given in this work. Firstly, an overview of anomaly detection is presented with emphasis on the issues about protocol anomaly detection. Then, a novel protocol anomaly detection model based on HMM is proposed. This method filters incoming TCP traffic by destination ports and then quantizes network flags into decimal numbers. These numbers are classified into sequences which are used as inputs of HMMs by TCP connections. Detection models based on HMM representing normal network behaviors are trained by Baum-Welch method. Finally, the models correctness and effectiveness is demonstrated by using forward method on MIT Lincoln Laboratory 1999 DARPA intrusion detection evaluation data set. Forward method is used here to compute the probability of a connection. Threshold K is designed to control detection rate. By comparing the probability with threshold K, this protocol anomaly detection model could find whether the traffic is normal or containing some sort of anomaly. Experimental results show that the model based on HMM has higher detection rates on attacks than the Markov chain detection method.