A study of sparse representation-based classification for biometric verification based on both handcrafted and deep learning features

Wright et al. [14], for the first time, put forward the SRC model and showed its significant improvement in face identification. Its success has largely boosted the research of biometric recognition based on sparse representation and collaborative representation [5]. A number of variants and extensions of SRC have been proposed in the last decade. Meanwhile, there are also many works that pay attention on the source of their discriminative ability [5, 19, 20, 25, 39].

A major direction is to explore its capacity to handle complex variations like illumination, pose, corruption, and occlusion. Yang et al. [40] proposed a Gabor-based SRC (GSRC) using Gabor features, which was shown to be more robust against illumination changes and pose mismatches. They also proposed a robust sparse coding (RSC) model by regarding the sparse coding as a sparsity-constrained regression problem [15]. RSC can effectively estimate the corrupted pixels and occluded regions and then exclude them from sparse representation in an iterative process. Zhou et al. [41] proposed to detect the contiguous occluded regions in test image using Markov random fields. Illiadis et al. [42] proposed a fast low-rank and iterative reweighted nonnegative least-squares algorithm, namely F-LR-IRNNLS, to address the problem of contiguous occlusions. F-LR-IRNNLS considers the error image is low-rank in comparison of image size, and it follows a distribution that can be described by a tailored potential loss function. Lai et al. [43] proposed a modular weighted global sparse representation (WGSR) method that divides an image into modules and sparsely encodes each module separately, and then dynamically combines their reconstruction errors based on their reliability for final classification. Lai et al. [44] proposed a collaborative patch framework using class-wise sparse representation (CSR-CP) to tackle the problem of uncontrolled training data. CSR-CP optimizes all patches together to seek a groupwise sparse representation by putting all patches of an image into a group.

Although SRC and its extensions have significantly improved the robustness of biometric identification, they are often criticized by the harsh requirements on the quality and the number of training samples per subject, and the poor efficiency in solving sparse optimization problem in large-scale scenarios. SRC requires the training samples per user which are sufficient and well-controlled to maintain its superior performance [14]. However, in real-world applications, the training data often contain a large number of identities but sufficient representative images for every identity cannot be guaranteed. To solve such an insufficient training samples problem, or saying under-sampled problem, a lot of effort has been made in the community. Deng et al. proposed several dictionary augmentation methods to enhance the representation ability of the gallery dictionary, including extended SRC (ESRC) [18], superposed SRC (SSRC) [19], and superposed linear representation classifier (SLRC) [20]. They take advantage of intra-class variation, class centroids, and the sample-to-centroid difference to construct the coding dictionary. Gao et al. [45] proposed a semi-supervised SRC (S³RC) using a variation dictionary to represent the linear variation of test sample and using a learned gallery dictionary based on Gaussian mixture model (GMM) to represent the non-linear variation. Jiang et al. [46] proposed a sparse- and dense-hybrid representation model based on a supervised low-rank dictionary decomposition/learning, aiming to alleviate the under-sampled problem and the uncontrolled training data problem simultaneously. Yang and Wang et al. paid attention on more fine-grained part-based methods [22, 23]. The face image is divided into multiple overlapping patches, centered around 5 facial keypoints and 16 regularly sampled facial points. A joint and collaborative representation is performed on the local dictionaries, each with an intra-class variation sub-dictionary, based on the local convolution or Gabor features for the final classification. The aforementioned methods and strategies have alleviated the problems of under-sampled and uncontrolled training data in small datasets to a certain extent, but their effectiveness in large-scale datasets remains to be tested.

Recall that the SRC methods use the training templates of all classes to construct coding dictionary, and thus, the computational cost of sparse optimization increases with the growth of dictionary scale [14]. Therefore, how to improve the efficiency is crucial in large-scale user applications. To alleviate this issue, Xu et al. [47] proposed a two-phase test sample representation method for face recognition. The first phase uses all of the training samples to represent the test sample using the more efficient L₂-norm based collaborative representation and select a limited number of “nearest neighbors” according to the representation ability of each training sample. Xu et al. [48] further improved the method using both the original and new generated ‘symmetrical face’ samples of a small number of classes that are ‘near’ to the test sample to represent and classify it. He et al. [49] proposed to filter the database into a small subset based on the nearest-neighbor criterion in a learned robust metric, and then perform nonnegative sparse representation-based classification with a small dictionary. All the above methods use a two-stage strategy that selects a small subset from the entire database in some efficient way and then performs SRC using the dictionary built with the selected data. Although compared with the one-stage SRC methods, they can substantially reduce the computational cost, the data filtering in the whole dataset are still very time-consuming in large-scale user applications.

More than 2 decades ago, Verlinde et al. [50] have proposed a one-to-many matching biometric verification method using a k-NN classifier. This is one of the pioneering attempts to consider non-target subjects in the test phase for verification. Cohort-based score normalization also takes advantage of non-target subjects, but serves the conventional one-to-one matching verification [51]. Nevertheless, the verification using non-target subjects and one-to-many matching did not receive much attention. Recently, inspired by the great success of SRC-based identification, SRC has also been introduced in the fields of the unimodal verification of speaker, face, and finger vein [27‐32], and the multimodal verification using face and ear [3].

In Ref. [27], GMM mean supervectors are used as features of an utterance to construct coding dictionary. The L₁-norm value of the coding coefficients associated with the claimed identity is used as genuine score, while such L₁-norm values for the other classes are imposter scores. Although, their experiments did not show improved performance of the proposed SRC-based verification, whose complementary information to the standard UBM-GMM classifier was clearly validated. Li et al. [28] built the coding dictionary using the i-vectors from total variability as atoms and evaluated three sparsity-based measures for speaker verification, including L₁-norm ratio (i.e., SCR), L₂ residual ratio, and a Bnorm L₂ residual (a regularized SCE measure). The Bnorm L₂ residual measure outperforms the other two measures in their experiments. The SRC-based verification approaches get inferior performance compared to the SVM classifier based on cosine similarity. However, improved verification results are achieved when combing sparsity-based scores and the SVM results. In Ref. [29], Kua et al. also investigated the i-vector-based SRC verification (iSRC) using L₁ constraint, L₂ constraint, and both constraints (Elastic net) in the coding optimization. The L₁-norm ratio is used as verification criterion, which was claimed to be superior to the other two measures proposed in Ref. [28]. They also validated that a small-size dictionary chosen based on column vector frequency can improve the verification accuracy and efficiency. Hasheminejad and Farsi [30] proposed to learn target, background, noise dictionaries with orthogonal atoms, and concatenated them together to construct overcomplete dictionary for speaker verification. The derived Bnorm L₂ residual scores are transformed to log-likelihood-ratio scores before decision. They reported better verification performance than iSRC.

Xin et al. [31] also utilized SCE as matching measure in finger vein verification and got a very low EER of 0.017% on a dataset with 600 fingers, which is also better than many competitive methods in their experiments. Shin et al. [32] performed sparse representation of the test color face image on each color channel of the color configurations, for example, and combined their class-specific reconstruction residuals (i.e., SCE) with Sum fusion rule for verification. The approach surpasses the one-to-one matching verification using LBP and Gabor features by a large margin of 12–22% EER on CMU Multi-PIE and Color FERET face datasets. However, such a system heavily relying on color channels would be sensitive to facial appearance variations, illumination and sensors in applications. And, the EER results of 1.89% and 2.79% they achieved are still very high for real-world applications.

In Ref. [3], Huang et al. performed sparse representation on the face and ear modalities, respectively. The multimodal verification based on the summation of SCE scores of the two modalities achieves about 0.2% EER on a multimodal dataset built with AR face dataset and USTB III ear dataset. However, the method is sensitive to the worst-case partial spoof attacks. Aiming to improve the anti-spoofing performance, they also proposed to use collaborative representation fidelity with non-target subjects to measure the affinity of the query sample to the claimed client. The resulting SCE scores and affinity scores of the two modalities are then combined in a stacked way to train an SVM classifier. The method was reported with promising anti-spoofing performance, and meanwhile, it achieves a good trade-off between verification accuracy and anti-spoofing.

Most of the studies of SRC-based verification are in the speaker verification community. Overall, the verification improvement brought by SRC-based verification is limited, but consumes expensive computation cost. This could be attributed to the large intra-class variations and the small inter-class variations of speech signals, which somewhat violates the two preconditions of SRC application. According to the existing literature, it seems that the SRC-based verification did not get much attention in the mainstream biometric community like face, iris, and fingerprint. In this paper, we are trying to uncover the limitations of SRC-based verification and the concerns about it in the community. We hope that our experimental results and findings will rekindle interest in SRC-based verification. In the end, we would also like to emphasize that the SRC-based verification is different from the approaches in Refs. [52‐54]. In these studies, sparse coding is used for feature extraction based on a learned dictionary and the verification still operate in the conventional one-to-one matching way. This will lose some of the benefits of competitive matching between the client and non-target subjects.

Assume that there are sufficiently enough well-controlled training samples for each class in the dataset with K classes. For simplicity, suppose all the classes have \(n\) training samples. Let \({\mathbf{A}}_{i} = \left[ {{\mathbf{a}}_{i,1} ,{\mathbf{a}}_{i,2} , \cdots ,{\mathbf{a}}_{i,n} } \right] \in R^{M \times n}\) be a matrix formed by the training samples of ith class, \({\mathbf{A}} = \left[ {{\mathbf{A}}_{1}^{{}} ,{\mathbf{A}}_{2}^{{}} , \cdots ,{\mathbf{A}}_{K}^{{}} } \right] \in R^{M \times N}\)(\(N = n \times K\)) is the dictionary composed of all training samples. Given a query sample \({\mathbf{y}}\), it can be represented by \({\mathbf{y}} = {\mathbf{A}}{\varvec{\alpha }}\), where \({{\varvec{\upalpha}}} \in R^{N}\) is the coding coefficient vector. If \(M < < N\), generally a sparsest solution can be sought by solving a \(L_{0}\)-norm optimization problemwhere \(\left\| \cdot \right\|_{0}\) denotes the \(L_{0}\)-norm, and \(\varepsilon > 0\) is a constant.

Solving \(L_{0}\) optimization problem in (1) is NP hard and extremely time-consuming. However, if the solution of \(L_{0}\) optimization problem in (1) spare enough, it is equal to the solution to the following \(L_{1}\)-norm optimization problem [14]:where \(\left\| \cdot \right\|_{1}\) denotes the \(L_{1}\)-norm. This problem can be solved in polynomial time by standard linear programming algorithms [61].

In our experiments, we use \(l_{1} - 1{\text{s}}\) optimization method [30] to solve sparse coding problem. Once \({\hat{\varvec{\alpha }}}\) is achieved, the class-specific sparse coding error (SCE) of ith class can be calculated using the coefficients associated with ith class as follows:where \(\delta_{i}\): \(R^{N} \to R^{N}\) is the characteristic function that selects the coefficients associated with the ith class.

The SRC and most of its extensions identify a query sample by sorting all the resulting SCE scores and then assign the class with the least SCE score.

Rather than examining all class-specific matching scores of all classes in dictionary, SRC-based verification can only calculate the matching score of the class associated with the identity claimed and then compare it with a predetermined operating threshold for an acceptance or rejection output. Figure 2 shows the flowchart of SRC-based unimodal verification. A client claims an identity and the corresponding sub-dictionary is used to build an overcomplete dictionary together with the sub-dictionary of non-target subjects.

Suppose \({\mathbf{A}}_{c} = \left[ {{\mathbf{a}}_{c,1} ,{\mathbf{a}}_{c,2} , \cdots ,{\mathbf{a}}_{c,n} } \right]\) is the sub-dictionary of the client claimed, \({\mathbf{A}}_{b}^{{}} = \left[ {{\mathbf{A}}_{1}^{{}} ,{\mathbf{A}}_{2}^{{}} , \cdots ,{\mathbf{A}}_{k - 1}^{{}} } \right]\) is the sub-dictionary composed of the gallery samples of the non-target subjects involved, or saying background dictionary, and then, the overall dictionary can be rewritten as \({\mathbf{A}} = \left[ {{\mathbf{A}}_{c}^{{}} ,{\mathbf{A}}_{b}^{{}} } \right]\). In our experiments, if no specific instructions, the coding dictionary \({\mathbf{A}}\) is composed of gallery samples of all subjects in each dataset. The SCE score associated with the claimed identity can be calculated as follows:

The superior identification performance of SRC and its extensions have validated that the SCE, as a distance measure, is a good candidate to measure the correlation between a query sample and a specific class. Thus, it is reasonable to use SCE for verification. Considering the binary classification in verification decision, the output is either acceptance or rejection, which can be denoted with 1 and 0, respectively. Given an operating threshold \(\theta_{sce}\), the verification rule with SCE can be written as

The sparsity concentration index (SCI) is a measure proposed along with SRC and SCE measure [14]. SCI measures how good the coding coefficient vector itself is in terms of localization. SCI is close to 1 when the query sample is encoded using only the dictionary atoms from a single class, while it is close to 0 if the coefficients spread evenly over all the classes. We refer the readers to Ref. [14] for its detailed formulation. SCI is often used to validate whether a query sample is a valid sample from the subjects in the coding dictionary, as a complementary measure with SCE.

Essentially, SCI depends on the class that contributes the most in sparse coding, whose value is the largest Sparsity Contribution Rate (SCR) among those of all classes in the dictionary. SCR reflects the participation degree of a specific class in representing the query sample. A larger SCR value indicates a higher possibility of the query sample belonging to a specific class. Therefore, SCR can also be used as a similarity measure for verification. The SCR score associated with the class claimed can be calculated as follows:

Apparently, \(\rho_{c} \left( {{\hat{\varvec{\alpha }}}} \right) \in \left[ {0,1} \right]\). The verification rule with a given threshold, \(\theta_{scr}\), can be written as

Figure 3 demonstrates the distributions of SCE and SCR scores obtained on AR face dataset. As for SCE, most genuine scores distribute in [0, 0.5], while the impostor scores concentrate around 1.0. On the contrary, almost all impostor scores of SCR are close to 0, while its genuine scores spread in a wide range. Compared with SCE, the overlap between genuine and impostor distributions is rather evident. This implies that the verification based on SCE should be better, which will be demonstrated in the section “Experiments”. The disadvantage of SCR could possibly originate from that (2) is solved by choosing \({{\varvec{\upalpha}}}\) to minimize the overall SCE but not the SCR [14].

Note that some variants of SCE and SCR are used for speaker verification in Refs. [28, 29]. However, Kua et al. [29] found that SCR is best in their speaker verification. SCE are also selected in Refs. [28, 31] for face and finger vein verification. Moreover, SCE and SCR have already been investigated in a variety of biometric identification applications [40, 55, 62]. SCE signals the representation ability of SRC, while SCR reflects the sparseness of coding vector. They are more general and representative, and thus more suitable for exploring SRC-based verification features in this paper.

In contrast to the conventional one-to-one matching verification, SRC-based verification conducts one-to-many matching between the query sample and the templates of the client and non-target subjects. SRC-based verification provides a competing mechanism, i.e., sparse coding optimization, which allocates class-specific sparsity-based matching scores to all classes in dictionary according to their correlations with the submitted data. Moreover, with the sparsity constraint, SRC generally allows only one or very few classes to get a convincing sparsity-based matching score, while the remaining classes get very inferior scores, as shown in the upper plot of Fig. 4. In other words, to be accepted, the genuine class needs to defeat almost all the non-target classes in the competing coding process. Therefore, an acceptance response made by SRC-based verification should be more convincing than those based on one-to-one matching. Overall, SRC-based verification not only examines the matching score obtained by the client, but also implicitly compares the correlations of the query data to the client and many non-target subjects, and thereby offers enhanced protection for identity security.

Figure 5 shows the SCE and SCR score distributions obtained on GT face dataset. We divide the genuine scores into two categories. The first category is the top 5 scores among all matching scores obtained in a verification process. The second category is the remainders. The distributions of two categories are outlined with red and blue lines in Fig. 5. It is quite clear that in both SCE and SCR cases, the top 5 genuine score distribution has a very trivial overlap with impostor score distribution. This means that a top rank usually comes along with a favorable genuine score, and vice versa. On the contrary, the genuine scores out of top 5 are so inferior that they are all close to the impostor distribution center. These results show that if the genuine class can defeat most of the non-target classes, SRC-based verification system will generally accept the verification request. Hence, the rank information among the client and non-target subjects is implicitly employed by SRC-based verification, embedding in the sparsity-based matching score. This implies that the sparsity-based matching scores obtained from competitive matching are more discriminative than the scores that come from one-to-one comparison, e.g., Euclidean distance and cosine similarity.

Generally, compared with non-target subjects, a genuine query sample (or feature) with good biometric quality is more similar with the client, who can hence win an eligible score in the competition, superior to a predetermined operating threshold. If the submitted biometric sample is unreliable, for example, it is captured from an intruder or has poor biometric quality, it is likely that none of the classes can dominate the sparse coding competition. Consequently, the coding coefficients will be spread evenly over all classes [14]. The client claimed can only get an inferior score and is rejected. This is often the case for the fake biometric traits used in spoof attacks if without sophisticated biometric fabrication. Besides the security improvement, SRC-based verification can also achieve significant advantage in verification accuracy in both unimodal and multimodal scenarios, compared with the conventional one-to-one matching verification. The experimental results will be presented in the section “Experiments”.

However, biometric sample quality often fluctuates with the illumination, pose, and appearance variants [33‐35]. Moderate data degeneration is inevitable in real-world applications. The submitted sample could probably be much different from the gallery samples of the claimed client, or even be somewhat similar to those of non-target subjects. Under these circumstances, the genuine client may fail to achieve a top rank in the competing sparse coding. As a result, the client will get an extremely inferior sparsity-based genuine score, as shown in the lower plot of Fig. 4. We can see in Fig. 5 that although distribution overlap between the genuine and impostor is not evident, there are a certain number of genuine scores spread near the impostor score distribution center.

Moreover, these genuine scores are so inferior that in real-world applications, it is impossible to accept the verification requests by tuning operating threshold, for the sake of avoiding high false accept rate. Figure 6 shows the ROC curves of SRC-based unimodal verification using SCE and SCR on all unimodal datasets. In the SCE case, they either get very high false reject rates in a wide range of FAR variation or their ROC curves almost do not drop at all after 10% FAR. In the SCR case, although the ROC curve on AR face dataset finally converges to 0% FRR, it is unacceptable for a 44% FAR in real applications. We call the phenomenon that the ROC curve cannot converge to 0% FRR or has a very long and flat tail along the FAR axis as FRR bottleneck problem. An evident FRR bottleneck problem will degrade user experience, and even makes the SRC-based verification unacceptable.

Due to the involvement of non-target subjects, SRC-based verification meets another challenge, that is, computational cost. Suppose that the atom dimensionality M in \({\mathbf{A}} \in R^{M \times N}\) is fixed, the complexity of sparse coding optimization depends on the number of atoms. For example, the empirical complexity of commonly used l1-ls optimization method to solve (2) is O(N^v) with v = 1.5 [15, 61]. In the applications with large-scale users, if using the gallery samples of all enrolled clients to build a coding dictionary, the computational cost would be prohibitively expensive.

Furthermore, it is well acknowledged that the increase of dictionary scale usually leads to accuracy degradation for SRC-based identification [36, 37, 39]. Likewise, SRC-based verification may confront the same challenge in the applications with large user scale. The more non-target subjects involved in sparse coding, the higher possibility to increase the distribution overlap in feature subspace between the sub-dictionaries of the genuine class and the non-target classes, i.e., \({\mathbf{A}}_{c}^{{}}\) and \({\mathbf{A}}_{b}^{{}}\).

As illustrated in Fig. 7a, the convex hull spanned by biometric samples is only an extremely tiny portion of the unit sphere \({\text{S}}^{M - 1}\), and the training vectors are tightly bundled together as a “bouquet” [63]. In the overall convex hull, the distribution interval among classes is very small, and many classes may overlap their neighbors to some degree. In reality, the appearance variations, pose, and alignment error will also aggravate the distribution overlaps. Accordingly, the more classes in dictionary may introduce more distribution overlaps. The inter-class separability between the genuine class and the overall non-target classes plays a critical role in recognition [20]. In the section “Experiments”, we empirically confirm the evident correlation between the inter-class separability and SRC-based verification accuracy.

Overall, SRC-based verification has to contend with the following challenges: (1) Once the genuine class fails to get a top rank in encoding the query data, it is very likely to result in extremely inferior genuine score. Consequently, the SRC-based verification may suffer from evident FRR bottleneck problem. If this problem cannot be resolved or avoided properly, SRC-based verification may not be suitable for some biometrics and the application scenarios that require both very low FRR and FAR. (2) Owing to the involvement of non-target subjects, the larger scale of the coding dictionary used, the more likely it is to degrade the verification efficiency and accuracy.

One may also notice that SRC techniques generally require multiple training samples per class for dictionary construction [14, 28]. This requirement is rather harsh and even impractical for some identification tasks. However, SRC-based verification is designed for the positive recognition scenarios where user cooperation is generally available [1]. A representative set of biometric samples per user can be captured in registration phase. If the gallery samples collected are not sufficient, there are still many ways to generate simulated biometric samples for users based on their enrolled data [64‐66]. The 3D face models [64] or generative adversarial networks (GAN) models [65, 66] learned from gallery samples can be used to generate samples with variants like pose and illumination. The dictionary augmentation skills like supplementing an intra-class variation dictionary can also be helpful to alleviate this problem [18, 20]. Besides, for the non-target subjects, SRC-based verification does not have to require their gallery samples with equivalent number and representative capability.

In the application scenarios with large user scale, if using gallery samples of all the enrolled subjects to build coding dictionary, it seems inevitable to degrade verification accuracy, and meanwhile significantly increase computational cost. In this subsection, we will first clarify the unnecessary use of a large number of non-target subjects to construct dictionary. Then, we propose a straightforward but effective strategy to shrink the dictionary via cluster analysis and random selection.

The non-target subjects used in coding dictionary play a critical role in SRC-based verification. Security improvement can be achieved through the one-to-many competitive matching among the client and non-target subjects. The more non-target subjects engage in the competition, the higher reliability of an “acceptance” decision can be achieved. However, the more non-target subjects involved also increase the intensity of competition, thereby increasing the likelihood of false “rejection” to genuine clients, while inevitably leading to higher computation burden. From these viewpoints, an excess number of non-target subjects are not only unnecessary, but can also cause negative effects.

To explicitly illustrate the risk of using excess non-target subjects for dictionary construction, we plot two toy examples in Fig. 7. Suppose there are 6 subjects, 4 training samples per class, and the feature dimension is 2, the convex hull is depicted in Fig. 7b. Considering a \(K = 6\) sparse \(L_{0}\)-norm sparse coding problem, the percentage of the K nonzero entries obtained by a class can reflect the probability of the query sample belonging to this class, similar with the SCR measure in \(L_{1}\)-norm optimization. Given a query sample of Class 1 on its distribution boundary, if randomly selecting half of the classes and using all their training samples as dictionary atoms, 3 training samples of Class 1 could possibly be used to represent the query sample, as illustrated in Fig. 7c. In this case, the genuine score is \({3 \mathord{\left/ {\vphantom {3 K}} \right. \kern-\nulldelimiterspace} K} = 0.5\). On the other hand, if using the training samples of all the classes for coding, the genuine score may become smaller like \({2 \mathord{\left/ {\vphantom {2 K}} \right. \kern-\nulldelimiterspace} K} = 0.33\), as shown in Fig. 7b. This score is much lower than the 0.5 achieved in Fig. 7c. It might not be a problem in close-set identification, since Class 1 still gets the top one rank. However, in verification, such a low score inclines to cause a false rejection by the comparison with a predefined operating threshold.

Considering the above observations and analyses, it is inadvisable to use a large number of biometric samples of the enrolled subjects to build a coding dictionary for verification. Here, we consider a simple dictionary shrinkage strategy via cluster analysis and random selection. The basic idea is that we first conduct cluster analysis on the training samples of all enrolled subjects, and then randomly select a few subjects from each cluster and use their training samples to construct an overcomplete dictionary with a limited number of classes. Cluster analysis is applied to avoid the worst-case scenario where all the selected subjects are concentrated in a tiny space and their distributions overlap heavily. To differentiate from the full dictionary with all the enrolled subjects, we call such a dictionary as small random dictionary.

In our experiments, all the query samples are used for testing, including those of the subjects selected for dictionary construction. The coding dictionaries for the selected and unselected classes are not the same but equivalent, that is, they share the same \({\mathbf{A}}_{b}^{{}}\). For example, 50 subjects are selected in PolyU dataset. In testing phase, the query samples of these classes are encoded based on the dictionary built with them. When the query samples are of the remaining classes, one of the 50 subject is replaced by the class claimed to construct dynamic coding dictionary. Hence, the resulting sparsity-based matching scores should be compatible. By conducting experiments on different small random dictionaries with the same scale on unimodal and multimodal datasets, we find that they consume much less time and can bring about better or comparable verification accuracy, compared with the full dictionary. This evidence supports that shrinking the dictionary by selecting non-target classes and training samples is feasible to avoid the heavy computational burden and recognition accuracy degradation on large-scale dataset.

The benefits of small random dictionary are summarized as follows. First and foremost, the smaller scale of dictionary means the lower sparse coding complexity and thus makes the SRC-based verification more efficient. Second, compared with large-scale dictionary, smaller scale dictionary is less likely to bring in more distribution overlaps among classes. Besides, the strategy of using small random dictionary is simple and requires no complicated training and preprocessing.

According to the analyses above, the SRC-based verification is rather suitable for the application scenarios where well inter-class separability is available. It is well acknowledged that in the multimodal feature space classes are better separated than in the unimodal feature space [1, 3, 4, 33, 34, 67]. Besides, it is much more difficult to spoof multiple biometric modalities than to spoof only one modality [4]. In this paper, we study the SRC-based multimodal verification with the combinations of face and ear, face, and palmprint, using Sum fusion rule.

The ear is located near the face, and can be captured along with the face using the same type of sensor or a single sensor at two times. Face detection can also help speed up ear detection by offering an ear region of interest. Most popular face feature extraction and classification techniques are applicable to the ear and palmprint. The recognition systems using ear and palmprint are also contactless. Besides, the ear has several appealing merits over the face: the ear has a stable structure with rich information, nearly unaffected by aging and expressions [68, 69]. Although there is a common impression that human ears are usually occluded by hair, it can be avoided via user cooperation in verification scenario. The studies in Refs. [33, 34] have already validated that the multimodal identification with face and ear can significantly improve the recognition accuracy and robustness. Compared with the face and ear, it is much harder to steal a person’s palmprint.

Suppose \({\mathbf{A}}^{f} = \left[ {{\mathbf{A}}_{c}^{f} ,{\mathbf{A}}_{b}^{f} } \right]\) and \({\mathbf{A}}^{e} = \left[ {{\mathbf{A}}_{c}^{e} ,{\mathbf{A}}_{b}^{e} } \right]\) are, respectively, the face and ear coding dictionaries. The SCE and SCR matching scores of the face and the ear can be calculated using (4) and (6), respectively. As shown in Fig. 8, the proposed SRC-based multimodal verification system first performs two independent sparse coding procedures, and then integrates the derived sparsity-based matching scores. Since the SCE or SCR of the face and the ear have similar distributions, we hence directly combine them without score normalization, empirically which brings no improvement in our experiments.

For convenience, let \({\text{s}}_{{}}^{f}\) and \(s_{{}}^{e}\) be the sparsity-based matching scores of two modalities. The multimodal matching score with Sum fusion is calculated as \(s = s_{{}}^{f} + s_{{}}^{e}\). The multimodal verification system makes decision with the similar rule in (5) or (7), according to the measure used.

Figure 9 plots the distributions of the multimodal SCE and SCR scores obtained on the MD I dataset. The overlap between the genuine and impostor distributions is trivial in both cases. Especially, both categories of the multimodal SCE show evident distribution centers that are far apart from one another. This implies a good robustness of the multimodal verification system. The detailed experimental results will be given in the section “Experiments”.

For convenience, we denote the SRC-based verification methods with SCE and SCR as SRC_sce and SRC_scr, respectively. The l₁_ls optimization [61] is used to solve the sparse coding problem. The unimodal verification method and multimodal verification method with Sum fusion [70] based on one-to-one matching and cosine similarity are used as unimodal and multimodal baselines. The multimodal SRC_sce and SRC_scr are also compared with the well-known multimodal methods, i.e., LLR [38] and SVM [71]. The multimodal SVM method fuses the matching scores of all the modalities in a stacked way and uses the RBF kernel with sigma of 0.25. They also use cosine similarity scores and are evaluated with tenfold cross-validation method.

In the experiments using ArcFace-based CNN features, the publicly available pretrained ResNet 50 model¹ is used to finetune for each modality. This model was trained on the MS1M-Arcface² dataset with the Arcface loss. Note that we revise the networks output to be a 200-D feature embedding, hence, we use the third-party dataset and some gallery samples of our datasets to separately finetune the networks. We use 2 gallery samples/subject of AR, 3 gallery samples/subject of GT, and a small subset of CASIA-WebFace to finetune the face model. The ear network is finetuned with 4 gallery samples/subject of USTB III, and 3352 ear samples of 300 subjects collected from college students. The palmprint network is finetuned with 3000 gallery samples of the remaining 300 subjects of the PolyU 2D&3D palmprint dataset. In the finetuning, the batch size is 32, initial learning rate is 0.001, weight decay is 0.0005, and momentum is 0.9. We train each model on one NVIDIA RTX 2080ti GPU card.

For each type of the sparsity-based matching measure, according to (4) and (6), given \(K\) classes in the coding dictionary, we can obtain one genuine score and \(K - 1\) impostor scores for a probe sample. For the SRC-based multimodal verification, we get 4400 genuine scores and 4400 × (50–1) = 215,600 impostor scores on MD I, 6083 genuine scores and 6083 × (79–1) = 474,474 impostor scores on MD II, 7000 genuine scores and 7000 × (100–1) = 693,000 impostor scores on MD III. As for the competing methods, we empirically select the best matching score from the comparisons of a probe sample and all training samples of a class, hence, the same numbers of genuine and impostor scores are available. All the experiments except CNN feature extraction are conducted on Matlab platform on a desktop with 3.3 GHz CPU, 64 GB RAM.