Abstract: Highlight event detection in soccer videos has high academic research value and wide market application prospect in the field of sport video semantic analysis. Based on the powerful expression of hidden conditional random field (HCRF) model in the expression and identification of semantic event, a fusion HCRF and affective arousal model (AAM) framework for highlight event detection is put forward. Firstly, through the analysis of the structural semantics of the wonderful event video, thirteen kinds of multi-modal semantic clues are defined to accurately describe the included semantic information of the wonderful events. Secondly, on the clustering foundation of the multi-modal semantic clues by concept lattice, time-domain features are added to establish an affective arousal model based on feature weight coefficient, and then the affective arousal value of the different kinds of highlights events is calculated. Finally, the above observed sequence is used as HCRF model input in the case of small-scale training samples, and a wonderful event detection HCRF model is effectively established based on the mapping relationship between the sequences of video semantic shots, affective arousal values and the highlight events. The inherent laws of the wonderful events are excavated from multiple dimensions like multi-modal semantic clues, video structure semantics, and affective semantics. The detection of wonderful events is simultaneously achieved by using the same HCRF model. Experimental results show the effectiveness of this paper.