Abstract: It is a common method to extract key frames using the unsupervised cluster algorithm. But the algorithm is sensitive to the initial number of the classes and the initial classification. It is problematic to predefine the absolute number of key frames without knowing the video content. An approach for two times clustering is presented. In the first time, the similarity distances of the consecutive frames in a shot are clustered into two classes so that the thresholds needed in the second time clustering process can be determined adaptively. In the second time clustering, all the frames in the shot are clustered using dynamic cluster ISODATA algorithm. Then the frame nearest to the center of its class is automatically extracted as one key frame in the shot. It is simple and effective with no need to predefine any threshold. Experimental results of many videos with different traits demonstrate the good performance of the proposed algorithm.