Abstract: With the development of Internet of Things technology, monitoring equipment has been widely deployed in public areas such as traffic arteries, schools and hospitals, shopping malls and supermarkets, and residential buildings. These devices provide a hidden safety and generate a lot of surveillance videos. Anomaly detection based on surveillance videos involves research efforts in image processing, machine vision, deep learning, and other related fields. In the paper, the intuitionistic description and anomaly detection of video anomalies are simply summarized, and some review articles did not cover the complete research scope about feature representation and modeling of the anomaly detection, as well as vague division. The research based on video anomaly detection is comprehensively analyzed. Firstly, the traditional classical and emerging video anomaly detection algorithms are classified and described from the aspects of anomaly detection feature representation and modeling. Then, we compare different algorithms based on distance, probability, and reconstruction, analyze the advantages and disadvantages of different models and characteristics of each model. Furthermore, we conclude the evaluation criteria of existing approaches and give the new accurate efficient evaluation index. Finally, we introduce the common datasets of surveillance videos on anomaly detection, summarize the detection effects of different algorithms on the common datasets, and discuss some challenges and future research directions in practical application.