Abstract: Anomaly detection of user behavior is now one of the major concerns of system security research. Anomaly detection systems establish the normal behavior profile of a subject (e.g. user), and compare the observed behavior of the subject with the profile and signal intrusions when the subject’s observed behavior differs significantly from the profile. One problem with anomaly detection is that it is likely to raise many false alarms. Unusual but legitimate use may sometimes be considered anomalous. This paper proposes a novel method for anomaly detection of user behavior, which is applicable to host-based intrusion detection systems using shell commands as audit data. Considering the property and the uncertainty of user behavior, the method obtains an event sequence with less variety of events after hierarchically merging shell command tokens into sets and then profiles the user’s normal behavior with a partly normalized co-occurrence matrix. In the detection stage, for event current sequence, a normalized co-occurrence matrix is constructed. Then the distances between these matrixes and the profile matrix are calculated according to the second matrix norm. Finally they are filtered with sliding windows and used to determine whether the monitored user’s behavior is normal or anomalous. The experiment results on datasets of Purdue University and SEA show that the proposed method can achieve higher detection accuracy, require less memory and take shorter time than the other traditional methods.