Abstract: Deep networks have been successfully applied to visual tracking by learning a generic representation offline from numerous training images. However, the features of the convolutional neural network abstraction algorithm are lack of spatio-temporal context information and the offline training is time-consuming. To tackle the above issues, an online convolution network tracking via spatio-temporal context is proposed, adopting the spatio-temporal context as the every order filter in convolutional neural network. Firstly, the initial target is normalized and the target confidence map is extracted. In the process of tracking, the spatio-temporal information is updated to obtain the spatio-temporal context model. The first layer utilizes the updated model to convolve the input and performs sliding window on the convolution result. The second layer convolves the fetch results by spatio-temporal model respectively, extracts the simple target abstract features, and then the convolution result of the simple layer is superposed to the deep level target expression. Finally, the target tracking is realized by the particle filter tracking framework. Our convolutional networks have a lightweight structure and perform favorably against several state-of-the-art methods on OTB-2013 and OTB-2015. As documented in the experimental results, the deep abstract feature extracted by online convolution network structure combining with spatio-temporal context model, can preserve related spatio-temporal information and then the tracking efficiency under complex background is improved.