Abstract: Moving object tracking is a critical issue of image sequence processing. In this paper, a moving object tracking algorithm based on location and confidence of pixels is proposed. Firstly, the moving objects are detected by combining the median background model in temporal domain with the minimum cross-entropy in spatial domain. Then the rectangle area of the objects are obtained, and at the same time an HSV color distribution model is used to measure the similarity between target rectangles and hypothetical rectangles. In this process, a weighting function based on location and confidence of pixels is presented to weigh the pixel values in the rectangle area of the tracking. The experimental results show that the algorithm is computationally efficient and robust to scale invariant, partial occlusion and interactions of non-rigid objects, especially similar objects.