Abstract: Based on the fact that human vision is especially sensitive to both the motion and edge of image, a new video object segment algorithm is proposed, which combines the inter frame difference and image gradient An approximate location of moving object can be obtained by using the difference between two continuous frames, then the exact boundary is defined according to the gradient of image, and finally an edge template is formed, from which a closed contour of VOP can be produced by using a shrinking active contour model At the same time, a template buffer is kept to remember the result of the previous frame, so as to make up for the incompletion of the inter frame diffrence of current frame The proposed algorithm is robust to the entire motion and local deformation of object, and can remove the uncovered background automatically as well as siege and segment multiple objects