Abstract: Retrieving similar video clips from large video database requires high query efficiency, precision, and recall, and it is a challenging problem in the field of multimedia information retrieval. In this paper, the video stream is segmented into segments by the visual similarity between neighboring frames, and the high-dimensional index structure, the vector-approximation file (VA-file), is adopted to organize the segments. Furthermore, a new similarity measure and query algorithm is proposed, which is based on restricted sliding window to improve the query accuracy. The proposed segmentation algorithm can efficiently represent the details of motion and the new similarity measure can fully take into account the temporal order among video segments. These properties well suit the retrieval of sports videos. Experimental results demonstrate that the proposed video retrieval method is efficient and effective.