Abstract: This paper presents an application layer anomaly detection method based on keyword sequences of application layer protocols. In this method, a hidden semi-Markov model is used to describe the behavior of a normal user who is using an application layer protocol, and the keywords as well as their inter-arrival times generated in using the protocol are used as the observation sequence on the user’s behavior. This method is divided into a training phase and a detection phase. In the training phase, the parameters of the hidden semi-Markov model are determined, through the forward-backward algorithm for the hidden semi-Markov model. In the detection phase, the average log likelihood of every observation sequence is calculated in real time. If a user’s behavior is abnormal while using some application layer protocol, the priority or the bandwidth of the packets belonging to the application will be reduced. In this way the user’s anomalous behavior will be restricted automatically. An experiment is conducted to validate this method, which is based on some data sets,including the DARPA dataset. The experimental results show that the model is effective in measuring the behavior of the normal users who are using some application layer protocol, and this method has high detection accuracy and low false positive ratio.