Abstract: Recently in some popular applications, such as video scene surveillance, long-term effective pedestrian tracking is the basis of these applications. Although the related technology of target detection and target tracking have a long history, how to achieve real-time and accurate pedestrian tracking is still an active research field and needs to be solved. At present, most pedestrian tracking methods only use hand-designed features to track or only use deep learning to extract features, which are not good ways to represent the features of the target because the use of one single feature will restrict the expression of the features. Therefore, multi-granularity hierarchical features are used in this paper to achieve more stable pedestrian tracking. This paper proposes an improved pedestrian tracking algorithm. The algorithm adopts the idea of multi-granularity, combines convolutional feature with bottom color feature, makes decision on the tracking result obtained by GOTURN, a tracking algorithm based on deep learning, and modifies the tracking result with target detection. This paper uses Pascal VOC data set for model training, and uses OTB-100 and VOT 2015 data sets for testing. The experimental results show that the tracking algorithm based on multi-granularity decision can track target pedestrians more accurately than a single tracking algorithm and the tracking accuracy is improved obviously.