Abstract: Shot is the basic unit of video analysis and retrieval. In order to perform shot boundary detection effectively and to classify different types of shot boundaries, a new feature-shot boundary manifold is proposed. Shot boundary manifold can be regarded as low dimensional local structure embedded in the high dimensional video data. Using this feature, a new post-refinement algorithm is proposed to eliminate the false positives and classify different genuine shot boundaries into four types: cut, dissolve, wipe, and fade. Experimental results demonstrate the effectiveness of the new shot boundary descriptor, and the good performance of post-refinement on shot boundary detection, and the good result of shot boundary classification.