Static–dynamic features and hybrid deep learning models based spoof detection system for ASV

Building the robust spoof detection system for Automatic Speaker Verification (ASV) is now an essential task, as the attention and demand for voice protected authentication systems is increasing in the users of smart devices. According to a survey users are curiously looking forward to use the speech driven authentication systems [1]. ASV system verifies whether the input speech signal is actually spoken by the authentic user or generated by the tricks by the imposter to gain access to the legitimate user’s account. With the availability of low cost voice sensors, and advanced research in mathematical and logical techniques for generating the synthetic speech, the number of spoofing attack types are also getting increased. Speech synthesis (SS), voice conversion (VC), replay, mimicry and twins attacks are the very potential spoofing attacks to these type of systems. SS attacked utterance is generated by the text to speech technique [2]. VC speech signals are generated by converting the imposter’s voice in to the legitimate user’s voice with the help of transformation functions [3‐5]. Replay attack are the one of the easiest form of attacks in which spoofed speech is the recorded voice signal of targeted user [6]. For mimicking the legitimate user’s voice, any professional manipulates his/her speech features. Twins attack is also a kind of mimicry attack [7, 8]. In some cases, twine siblings are able to get access to each other’s voiced locked accounts [5, 9]. SS and VC attacks can be injected via the channel into the system. Hence, these attacks are named as Logical Access (LA) attacks [9]. The replay, mimicry and twins attacks are inserted by the microphone into the system. Hence, these attacks known as physical access (PA) attacks. Performance of ASV systems is greatly affected in the presence of these spoofing attacks [10]. Various speech corpora have been proposed enriched with different kind of spoofing attacks. For instance, ASVspoof 2015 data includes SS and VC attacks [11], ASVspoof 2017 dataset includes only replay attack [12], Yoho dataset includes mimicry attacks [13], etc. The recently proposed ASVspoof 2019 dataset includes SS, VC and replay attacks, however, in two sets. This paper presents an initiative of putting all kind of attacks into a single dataset.

Along with attacks consideration, the robust designs of frontend and backend of an ASV system can become a preventive shield for spoofing attacks. Frontend of an ASV system uses a speech feature extraction technique to extract useful information form the recorded speech signal. Features of cepstrum domain that are Mel Frequency Cepstrum Coefficients (MFCC), Inverse Mel Frequency Cepstrum Coefficients (IMFCC) [14], Linear Frequency Cepstrum Coefficients (LFCC), Constant Q Cepstrum Coefficients (CQCC), etc. have performed remarkably well for the spoof detection tasks, and for speech and speaker recognition tasks as well. These techniques can model the human vocal tract and human auditory system very well [15‐17]. Human ear is proved to be deaf for the phase factor of sound. However, utilization of this factor for frontend development of speech driven devices [18, 19] can be done by using All Pole Group Delay Function (APGDF), Modified Group Delay Function (MODGDF), etc. Both static and dynamic coefficients of speech features deliver the information of context and speaker specification information. These coefficients are passed to the backend spoof detection model. CQCC features are specially designed for spoof detection tasks proposed in ASV systems of [20, 21] and it is claimed that, these features perform better than Instantaneous Frequency Cosine Coefficients (IFCC), MFCC, Epoch Features (EF). The proposed work in this paper also exploits a hybrid of static and dynamic CQCC features for developing the frontend. Also, it presents performance comparison of static and static–dynamic CQCC features by using Long Short Term Memory (LSTM) with Time Distributed Wrappers model at the backend.

Various machine learning techniques Gaussian Mixture Model (GMM), Hidden Markov Model (HMM) [22‐24], Convolutional Neural Network (CNN), Long Short Term Memory (LSTM), etc. are playing crucial role for classification tasks even in speech based systems [25, 26]. In case of ASV system, backend classification model takes the speech features as input and classifies the signal as spoofed or bonafide after analyzing the speaker specific information in them. In the initial research, GMM was used effectively as the backend model [27]. As the deep learning algorithms are getting improved day by day ASV community has started to use CNN and LSTM models [28‐30]. In various speech and speaker recognition tasks, LSTM-based deep learning models are performing better than the other models. However, CNN models are also giving satisfactory results [31‐33]. Also, different arrangements of frontend and backend models can bring smoothness and accuracy to spoof detection task.

The rest of the paper is organized as: second section discusses the related work then third section of the paper discusses the proposed method, the experimental setup details and results are presented in fourth section, fifth section explains the performance analysis of proposed models and systems then sixth section compares proposed systems with existing systems and seventh section concludes the proposal with dropping some light on future directions.

This section discusses the related works in this area. Literature is enriched with the experiments on various feature extraction techniques of audios at frontend and different classification models at the backend. Research done by Valenti et al. [34] discusses an approach with end to end speech signal passing to an evolving Recurrent Neural Network (RNN). System used in their work is designed with RNN and neuroevolution of augmenting topologies. The proposed work considers replay attack particularly.

The review done by Kamble et al. [35] presents a wide analysis of many existing ASV spoof systems from the perspective of ASVspoof challenge. Lai et al. [36] proposed Attentive Filtering Network based and ResNet classifier based system to detect replay attacks. The proposed attention-based filtering approach is used to improve feature representations. The proposed work used ASVspoof 2017 Version 2.0 dataset to attain a very low Equal Error Rate (EER). The authors claimed an improvement of about 30% over the existing ASVspoof 2017 enhanced baseline system.

ASVspoof 2019 challenge puts the three different types of attacks in one dataset and presents baseline models with LFCC and CQCC features at frontend and GMM at the backend [27]. Chettri et al. [10] trained various deep learning backend models and tested them with different features extraction approach in front end. These backend models are further combined to get three ensemble models, where all the systems were tested for physical access and logical access attacks.

Recently, Dua et al. [30] also proposed the ensemble approach using LSTM based deep learning models at the backend, and three different feature extraction techniques Constant Q cepstral coefficients (CQCC), inverse Mel frequency cepstral coefficients (IMFCC) and MFCC at the frontend. The author claimed that their proposed ensemble model with CQCC features outperforms some already existing proposed ASV systems.

Motivated by these works, the proposed work in this paper compares performances of different deep learning models at backend by using them with static–dynamic CQCC features at frontend. The implemented work of this paper has also used combination of LSTM and CNN models for development of the backend. Also, two two-level spoof detection systems for ASV by using static–dynamic features at frontend are implemented. The first system does voting protocol based implementation by using CNN, LSTM models at first level and LSTM with Time Distributed Wrappers model at second level. The second system uses LSTM model for user identification at first level and LSTM with Time Distributed Wrappers for verification at the second level. These systems can bring new insights in the development of spoof detection methods for ASV.

This section of the paper discusses the architecture of the proposed ASV system. Figure 1a shows the frontend and backend arrangement that has been used for comparison of static CQCC and static–dynamic CQCC features in the implemented ASV system. Speech signals taken from the dataset are applied to the frontend where static CQCC features are extracted with the general process of extraction and static–dynamic hybrid features are extracted with the proposed methodology. Then these features along with the labels from the dataset are applied to the backend model that runs the classification. These classification results are useful for feature comparison. Figure 1b shows the frontend and backend arrangement that has been used for comparison of various deep learning models by keeping static–dynamic CQCC features at frontend. Frontend used in this arrangement is the best performing feature extraction technique from the feature comparison. Speech signals and labels are the part of same dataset in whole arrangement. Backend here has all the proposed models for spoof detection and single model for speaker identification task. At the backend all chosen model are trained and their performances are analyzed. Systems of Fig. 2 are the arrangements of models from Fig. 1. Figure 2a shows the block diagram for the voting protocol based two-level spoof detection system. This system classifies the speech signal according to the voting protocol that is implemented with the help of level 1 and level 2. Level 1 applies analysis on the input that is further analyzed at level 2 as per the protocol to declare the decision. Figure 2b gives the block diagram for the two-level user identification and verification system. This two stages arrangement makes the use of speaker identification model at stage 1 result of which is passed to stage 2. Stage 2 uses the user identification and verification protocol along with chosen backend model to declare the classification result.

The following is the pointwise contribution of the proposed work and following subsections discuss each component in detail.

This section of the paper deals with the experimental details for implementation of the proposed ASV system. The frontend feature extraction is implemented by using Octave on Linux Operating System. The training, development and evaluation of backend models are done with Anaconda platform on Windows operating system. All the used audios and labels are taken from training, development and evaluation sets of AllSpoofsASV Dataset. During training the deep learning models, Python’s inbuilt features are used for weight updation, that is backpropagation algorithm and loss functions are used. For the two class classification problems, binary cross entropy loss is used as the loss function. It finds out the probability or score for an utterance between zero and one. Categorical cross entropy loss function is used as loss function for multi class classification of user identities (specifically in the training of M₄).

A learning rate is required for iterative updation of weights during the training process. In the proposed work, ADAM (Adaptive Momentum) optimizer algorithm is used to achieve the adaptive value of learning rate [43, 44]. It combines the advantages of Adaptive Gradient Algorithm (AdaGrad) and Root Mean Square Algorithm (RMSProp). AdaGrad defines the learning rate for each parameter to improve the performance of model sparse gradient, whereas RMSProp makes the use of average of latest values of gradients of weights. ADAM algorithm passes both the gradient and square gradient to the exponential moving average function. For heavy models and large size of datasets, it can solve practical problems efficiently [43‐45]. System arrangement for different comparisons and analysis are discussed later in this section.

The performance of the proposed architectures and systems are evaluated with the help of two evaluation measures Equal Error Rate (EER) and Percentage Accuracy. Spoof detection systems are evaluated by using EER and user identification system is evaluated by percentage accuracy. EER is the equal value of False Acceptance Rate (FAR) and False Rejection Rate (FRR) [27, 28], where FAR is ratio of number of spoofed utterances having score more than or equal to the threshold Ψ to the total number of spoofed utterances and FRR is ration of the number of bonafide utterances having score less than the threshold Ψ value to the total number of bonafide utterances. The mathematical representation FAR and FRR is given by Eqs. (5) and  (6), respectively. EER aims to calculate the FAR and FRR with the help of threshold Ψ. For the equal values of these parameters, it declares the EER for the system.

Percentage accuracy is calculated with the help of correct predictions and total number of input samples to be checked. Mathematical formula of percentage accuracy is given by Eq. (7).

In this case, the division of total correctly predicted user samples by the total number of user input samples is multiplied by 100.

This section presents the performance and comparison results of all systems discussed in third section. For obtaining the results, the proposed work uses the procedure adopted by state of the art works of [10, 15, 26]. As described earlier in “AllSpoofsASV dataset” section, the dataset used by the proposed system is already divided in training, development and evaluation sets. Therefore, it is not required to partition the dataset in ratios for training, development and evaluation samples. For evaluation in case of ASV systems, EER is the used evaluation protocol that is applied on the classification results of the model for spoof detection task [10, 15, 26]. Models for this work have been trained five times with the training set, and for each trained model development set is applied. Network parameters have been tuned for all the systems to obtain stable parameters. On the development results, EER evaluation protocol is applied and accuracy of the model is verified. Mean of all five development set test results is considered to show in presented tables. Evaluation set is applied on the model when it becomes stable after all training passes and EER is calculated for the classification result. Protocols of systems one and two are applied with the evaluation set performances of models. For the task of speaker identification, percentage accuracy is calculated as evaluation measure on development set results using five-fold validation approach. It is also evaluated for evaluation to check the performance.

This section compares the performances of the proposed systems, System_1 and System_2, with some of the existing systems from the literature. Chettri et al. [10] have designed three Ensemble systems (E1, E2 and E3) made up of different classical and deep learning models, where the ensemble system E1 performs the best among them. Cai et al. [15] have trained ResNet deep learning model with CQCC, LFCC, IMFCC, Short Term Fourier Transform (STFT) grams, and Group Delay (GD) gram features. However, it is trained only for replay attack. Kumar et al. [26] have trained a Time Delay Shallow Neural Network (TDSNN) with CQCC, IMFCC, Linear Frequency Band Cepstral Coefficients (LFBC) and LFCC features for SS, VC and replay attacks. ASVspoof 2019 challenge has provided a GMM model trained with LFCC and CQCC features at frontend for SS, VC and replay attacks [27]. Jung et al. [46] has trained a Deep Neural Network Model with 7 spectrograms, i-vectors and raw waveforms only for replay attack detection. Table 6 shows the comparison of these systems with proposed systems of this paper. Although, some systems from literature seem to be good for detection of a particular attack type. However, proposed system is also performing good for the detection of all three kinds of spoofing attacks in one run.

Undoubtedly, the ASV systems are highly exposed to spoofing attacks. However, their performance is fine enough that industry is attracted to use them in practical applications. Initiative to design a single dataset can provide new insights to the spoof detection task. AllSpoofsASV Dataset, a variation of ASVspoof 2019 dataset, is a small step towards this. Combination of different feature coefficients with hybrid deep learning models can help in development of robust ASVs. This paper shows that a combination of static and dynamic CQCC performs better with LSTM models than only static features. Also, comparison of results shows model LSTM with Time Distributed Wrappers (M₁) outperforms the models LSTM (M₂) and CNN (M₃), when evaluated by Equal Error Rate (EER). However, the two-level voting protocol based spoof detection system System_1 that uses M₂, M₃ at level 1 and M₁ at level 2 performs best of them all. As model LSTM (M₄) provides satisfactory performance it can be used particularly for speaker identification with spoof detection. Also, two-level spoof detection system with user identification and verification System_2 that uses M₄ at stage 1 and M₁ at stage 2 performs better than System_1. However, it is limited to limited number of users. Using it for public domain or an organization with more and variable number of speakers will increase the complexity and requirement of storage space for the system. For future work, more attacks like twins and mimicry should can be added into the dataset, and more hybrid possible combinations of features and deep learning models can be exploited. Considering the importance of the spoof detection in ASV, more efficient and complex structures like VGG-family of deep learning models can also be used as future extension of the proposed work.