Abstract: Speaker recognition realizes a simple and fast biometric authentication method which is non-contact, not easy to forge and can be authenticated remotely, However, it is not completely secure, as in the process of authentication, storing user data in a third party brings many security and privacy concerns. In order to mitigate the challenge, the template protection of speaker recognition problem is studied based on identity vector (i-Vector) and linear discriminant analysis (LDA), and an improved random mapping technique is proposed, which is used to randomize the voiceprint features. Then we construct a template protection scheme for speaker recognition, which allows users to register in random domain and to complete speaker recognition. Finally, we use the open Chinese speech data set (AISHELL) to simulate the proposed scheme. The results show that the scheme does not significantly affect the accuracy of voiceprint authentication, and realize confidential comparisons of voiceprint templates, which effectively ensures the security and privacy of voice data in speaker recognition.