Abstract: Realtime human-computer interaction(HCI) based on hand gestures plays an important role in both theory and application of virtual reality. Since the 3D position of fingertip can be tracked and located through stereovision technology with two cameras, the real-time interaction between fingertip and 3D objects in virtual world can be achieved. The proposed method in this paper can be widely used to implement 3D interaction for augmented reality based video games. In this paper, an improved BGS algorithm, which combines threshold decision with GMM, is presented to identify the hand region. The method can get hand region in different lighting conditions, and avoid the interference of cast shadow of hand. The fingertip can be determined with contour Kvector and distance between the hand region center and the candidate fingertip position. Compared with the general fingertip algorithms, the algorithm presented can get fingertips in unfriendly hand foreground segmentation. After calibrating two cameras, the authors get origin position in world coordinate relative to marker center, then 3D position of fingertip can be reconstructed by camera parameters and the fingertip positions in two images taken by the two cameras. Finally, Kalman filter is employed in 3D space to smooth the fingertip trajectory and predict the range of fingertips. Experimental result shows the efficiency of the algorithm.