As mentioned earlier, prior knowledge of the pose of a face is an essential information in many face recognition techniques. It is often beneficial if the pose angle of the input face image can be estimated before recognition such as in modular PCA (MPCA) (Pentl et al., 1994) and eigen light-field (Gross et al., 2004). There are many efforts for automatic pose estimation in the literature. As the focus of this paper is on face recognition and not pose estimation, an interested reader is referred to Murphy-Chutorian and Trivedi (2009) and Ding and Tao (2016) as good surveys on face pose estimation.

It is obvious that two face images from different identities in the same pose are visually more similar than two face images of an identity in different poses. This can be used as a clue for face pose estimation aim, a face image in a specific pose can be estimated by a linear combination of face images of other subjects in the same pose. The proposed pose estimation method is based on the assumption that sparse representation coefficients of different identities in the same pose are closer than representation coefficients of images of the same identity in different poses. Therefore, using the sparse representation of unseen face image over training images of the same pose and repeating this for all poses, one can estimate the pose based on minimizing the reconstruction error on different poses. A similar idea has been used in Yu and Liu (2014) where it is assumed that a face image in a specific pose cannot be approximated by a combination of face images in other poses.

Suppose there are P classes of different poses Ap , p=1,…,P . The p-th class Ap=[ap1,ap2,…,apnp]∈ℜ(dnp) is called the view dictionary of pose p that has np training face images from different identities in this pose and d is the dimension of each face image. Based on SR theory, every unseen face image y∈ℜd is expected to be expressed as a sparse representation of images in matrix Ap in a particular pose. The sparse coefficients of image y over the view dictionary Ap is the xˆp vector that can be obtained as follows: (3) xˆp=argminxp‖y−Apxp‖22+λ‖xp‖1, where λ is the regularization parameter as before. Therefore, face image y is reconstructed based on different view dictionaries. The view dictionary that reconstructs the face image with minimum error determines the pose of the face image y. In other words, the pose of face image y is estimated based on minimizing the reconstruction error among all view dictionaries: (4) pˆ=minp‖y−Apxp‖2,p=1,…,P, where pˆ is the estimated pose. Actually, this shows that Apˆ is the best view dictionary that can reconstruct the input face image from a linear combination of its set of face images. The proposed pose estimation algorithm is summarized in Algorithm 1. Figure 2 represents an example of the proposed pose estimation method where there are seven different poses (seven view dictionaries).

Figure 2(a) shows the input face image on the top and the 7-th reconstructed face images with respect to seven view dictionaries below that. As it is obvious, reconstructed image from the last dictionary (A7) seems to be the most similar one to the input face image, where the reconstruction error plot in Fig. 2(b) confirms this consequence. Thus, the input face image is supposed to be in the last pose.

The proposed pose estimation method has some advantages over many other pose estimation methods. First, there is no assumption on the number of training face images in each pose and view dictionaries can have a different number of atoms. Also, no feature selection or 3D model of the face is required for pose estimation, so no image registration and heavy computation is used. However, the main shortcoming of the proposed pose estimation method is its accuracy drop in small pose intervals which will be discussed more in Section 5.2.

In many face recognition methods, one of the key steps for achieving multi-pose face recognition is pose normalization or virtual frontal view generation. Obviously, a frontal face image contains the most details of the face which are beneficial for face recognition, compared to a non-frontal face image. In order to compensate for the loss of details in non-frontal views, one can try to generate a virtual frontal view from a non-frontal view. In this paper, this task is formulated as a general prediction framework which predicts a mapping from each non-frontal view to the frontal view, where the mapping is identity-independent. The purpose of this mapping is to estimate a frontal face image bˆ1∈ℜd given its non-frontal face image bp∈ℜd in pose p. Modelling the virtual frontal view generation with linear mapping is as follows: (5) b1=Vp(bp)=Wpbp, where Vp(.) is the linear mapping function and Wp is the linear mapping matrix for pose p. Linear mapping function Vp(.) can be achieved via a learning process. GLR and LLR (Chai et al., 2007) are general least square problems that use regression-based methods to find a good mapping. Another idea to find the mapping function is introduced in LSRR (Zhang et al., 2013) which assumes that the face images of one identity observed from different views share the same sparse representation coefficients over different view dictionaries. In other words, suppose f1 and f2 are two face images of one identity in poses p1 and p2 , respectively. The sparse representation coefficients of these two face images over view-dependent dictionaries related to p1 and p2 poses are supposed to be similar. Therefore, if sparse representation coefficients of a non-frontal face image of an identity are available over its view dictionary, these coefficients can be used to generate the virtual frontal view of that identity, using the frontal view dictionary. Consequently, considering face images of the i-th identity, we have the following set of equations: (6) b1i=A1xi+e1⋮bpi=Apxi+ep⋮bPi=APxi+eP, where Ap is the view dictionary of pose p, bpi is the face image of identity i in pose p and ep is the reconstruction error in pose p. The sparse representation coefficients xi are shared among all the P views of the i-th identity. These equations say that the face image from pose p can be generated from sparse representation coefficients xi with the corresponding view dictionary Ap . Therefore, the key of virtual view generation lies in the recovery of the sparse representation coefficients xi . The idea of sharing the sparse representation coefficients among different poses somehow reminds the idea used in Prince et al. (2008) where the authors assumed a face manifold and an identity space (latent space) and declared that the representation of each identity does not vary with pose. As another example, one can mention the research done in Sharma et al. (2012) which aims to find the sets of projection directions for different poses such that the projected images of the same identity in different poses are maximally correlated in the latent space.

Based on the discussion above, given training samples arranged in different view dictionaries Ap (p=1,…,P) , the non-frontal to frontal mapping function for input face images in pose p ( bp s) can be obtained by first finding the sparse representation coefficients of bp on view dictionary of pose p, then utilizing these coefficients with the frontal view dictionary A1 as follows: (7) xˆp=argminxp‖bp−Apxp‖22+λ‖xp‖1, (8) bˆ1=Vp(bp)=A1xˆp, where xˆp is the best vector of sparse representation coefficients of bp over view dictionary Ap , λ is the regularization parameter as mentioned before and bˆ1 is the virtual frontal view corresponding to the non-frontal view bp .

It is worth noticing that the mapping in Eq. (8) is based on two parameters, view dictionaries and sparse representation coefficients. As the sparse representation coefficients are obtained via an optimization problem based on a view dictionary, one of the factors that play an important role in high accuracy mapping is the selection of view dictionaries. These dictionaries can be simply made form training images of each pose or can be learned more effectively in a dictionary learning process. As training images are accompanied by identity label, using them as view dictionary atoms might not successfully generate a face image from a new identity. In other words, identity-independent view dictionaries are expected to be more efficient in generating face images from new identities. Therefore, one of the key steps for increasing the accuracy of the proposed method in obtaining the sparse representation coefficients and generating the virtual frontal view is to learn desirable identity-independent view dictionaries. The next subsection explains a supervised dictionary learning process to learn Ap s as efficient as possible.

Suppose that bp and b1 are two face images of one identity in non-frontal pose p and frontal pose 1, respectively, xˆp is the sparse representation of bp over view dictionary Ap , and xˆ1 is the sparse representation of b1 over view dictionary A1 . As mentioned in the previous section, view dictionaries Ap and A1 are called desirable if the sparse representations xˆp and xˆ1 are the same or at least close enough. So, the aim of this subsection is to learn view dictionaries that share similar sparse coefficients for face images of one identity in different poses. This is achieved via a supervised view dictionary learning process. By concatenating the P equations in Eq. (6), while omitting the identity parameter i for simplicity, we have: (9) b1⋮bp⋮bP=A1⋮Ap⋮APx+e1⋮ep⋮eP→B=Ax+esubject to‖x‖1⩽C, which states that the P views, when concatenated together, should have the same sparse representation with respect to the concatenated view dictionary. Given the training dataset {bpi}p=1,…,Pi=1,…,I , where i is the index for identities and p is the index for poses, the training set is rearranged by concatenating the P views of each identity in {bi}i=1,…,I vector, where bi=[b1ib2ibPi]T∈ℜ(dp×1) and B∈ℜ(dP×I) is the matrix made from concatenating face images of I different identities. Now, the view dictionaries can be learned via the following minimization problem: (10) ⟨Aˆ,Xˆ⟩=argminA,X ∑i=1I‖bi−Axi‖22+λ‖xi‖1, where Aˆ=[Aˆ1T,Aˆ2T,…,AˆPT]T∈ℜ(dP×K) is the learned dictionary and Xˆ=[xˆ1,xˆ2,…,xˆI]∈ℜ(K×I) is the sparse coefficients matrix whose column i is the sparse representation vectors of the training samples in bi and K is the dictionary size. Eq. (10) aims to jointly find the proper sparse representation coefficients and the dictionary. It describes face images from the ith identity ( bi ) as the sparsest representation xi over dictionary A. After Aˆ is learned, the view dictionaries {Aˆp}p=1,…,P are obtained by splitting Aˆ into P parts, e.g. view dictionary of pose p ( Aˆp ) can be achieved by separating d rows of Aˆ that are corresponding to the p-th pose (rows pd−d+1 to dp ). Figure 3 demonstrates these matrices and the dictionary learning process visually.

In order to properly choose the dictionary size K, it is worthy to remind some points on dictionary characteristics. Dictionary A∈ℜ(dP×K) is considered undercomplete if K<dP or overcomplete if K>dP dP$]]>. When ( K=dP ), dictionary is considered as a complete dictionary. From a representational point of view, a complete dictionary does not help in any improvement and is neglected. Undercomplete dictionaries are strongly related to dimensionality reduction. Principal component analysis is a famous example of this case where dictionary atoms have to be orthogonal. However, putting orthogonality constraint on dictionary atoms limits the choice of atoms which is the main disadvantage of undercomplete dictionaries. On the other side, overcomplete dictionaries do not have the orthogonality constraint, therefore, they allow for more flexible dictionaries and richer data representation (Elad, 2010).

Although all view dictionaries can be learned simultaneously using Eq. (10), the learning process will be impractical for large dictionary sizes or high dimensional data. Consider a situation where each identity has images in 10 poses and each image has about 1000 pixels (a small 30×35 face image), so each column of dictionary has about 10000 entries. If the dictionary size is adjusted to 1000 atoms, the size of dictionary will be 10000×1000. Doing computation on a dictionary of this huge size is impractical because of memory and computational limitations. To dominate this problem, in this paper, pairwise dictionary learning is proposed, where each view dictionary is learned separately. In other words, in order to learn the view dictionary for pose p, the training matrix will be Bp∈ℜ(2d×I) where each column of Bp is [bpib1i]T∈ℜ(2d×1) (face images of frontal pose and images of non-frontal pose p). In this case, the optimization in Eq. (10) results in Aˆ∈ℜ(2d×K) where the first d rows of Aˆ can be considered as learned view dictionary for pose p. It should be noted that view dictionaries are not learned simultaneously in pairwise dictionary learning. However, as training images in frontal pose are the same for learning all view dictionaries, it is expected that describing face images of one identity in different poses by these view dictionaries has similar sparse representation coefficients. Figure 4 shows the effect of dictionary learning on sparse representation of three face images of an identity in different poses. The first column shows the sparse coefficients when dictionary atoms are simply the training data in different poses while the second column shows sparse coefficients obtained based on learned dictionary. As expected, the representation coefficients of the three images shown in the second column are more similar compared to the ones in the first column. This observation confirms the effect of dictionary learning on unifying the sparse representation coefficients of face images of one identity in different poses. Also, the figure depicts the increase in sparsity of coefficients after dictionary learning, which is another aim of dictionary learning. Getting back to dictionary learning process in Eq. (10), several dictionary learning methods have been proposed since now that can be divided into two groups: 1) unsupervised dictionary learning methods such as MOD (Engan et al., 1999) and K-SVD (Aharon et al., 2006) and 2) supervised dictionary learning methods such as SDL (Mairal et al., 2009) and LCKSVD (Jiang et al., 2013). The K-SVD method is introduced to efficiently learn an overcomplete dictionary and has been successfully applied to image restoration and image compression. K-SVD focuses on the representational power of the learned dictionary, but does not consider the discrimination capability of it. LCKSVD (Jiang et al., 2013) is a supervised extension of K-SVD that uses supervised information (labels) of training samples to learn a compact and discriminative dictionary. As LCKSVD has proved itself as a successful supervised dictionary learning method, it has been used here for learning view dictionaries. The objective function defined by LCKSVD is as follows: (11) ⟨Aˆ,Xˆ,Tˆ⟩=argminA,X,T‖B−AX‖F2+α‖Q−TX‖F2s.t.∀i,‖xi‖0⩽C, where ‖.‖F is the Frobenius norm and B, A and X are training data, dictionary and sparse coefficients matrices, respectively. Q=[q1,q2,…,qI]∈ℜ(K×I) is the discriminative sparse code of training samples in B and is initialized based on the labels of training samples and desired labels of dictionary atoms. For example, if i-th atom of dictionary has the same label as j-th training sample, then Q(i,j)=1 , else Q(i,j)=0 . T is a linear transformation matrix and the term ‖Q−TX‖F2 enforces the sparse coefficients X to approximate the discriminative sparse codes Q. This term enforces the samples from the same class to have very similar sparse representations. The first and second terms of Eq. (11) are the reconstruction error and the discrimination power, respectively, where α controls the contribution between these two terms. The implementation of LCKSVD is available by the LCKSVD authors and is used in this paper for solving Eq. (11). For more details on LCKSVD, we refer the interested reader to Jiang et al. (2013).

After obtaining view dictionaries, virtual frontal view generation can be done by first estimating the pose pˆ of the input face image y by Algorithm 1, then finding xˆpˆ as the sparse representation of y over the learned view dictionary of the estimated pose. Finally, the virtual frontal view of y is generated by multiplying the sparse representation xˆpˆ to the learned view dictionary of frontal pose Aˆ1 . The algorithm of virtual view generation is summarized in Algorithm 2 and an example of virtual view generation is shown in Fig. 5.

The previous subsection explained the proposed virtual frontal view generation algorithm which was based on supervised dictionary learning. This subsection completes the previous steps by recognizing the generated frontal view. As mentioned earlier, SRC method has shown superior performance on frontal view face recognition, therefore, in this paper, SRC is used as the classifier in the recognition step. The overall view of the 3 steps of the proposed method is shown in Algorithm 3.

CMU-PIE and FERET databases contain a large number of face images in different illumination, viewpoints and expressions. CMU-PIE database has 68 identities who were imaged under 13 different poses, 43 different illumination conditions and 4 different expressions. Figure 6 shows the variation of poses in the CMU-PIE database, images are within −90° to +90° with ±22.5° interval in yaw and about ±20° in pitch. FERET database contains more than 1000 identities in different conditions. From them, 200 subjects have all 9 different pose variations within ±60° in yaw (0° in pitch). Specifically, the poses are ±60°, ±40°, ±25°, ±15° and frontal pose 0°. Figure 7 shows the variation of poses in the FERET database. In our experiments, 5 poses of CMU-PIE database are used in 90°, 67.5°, 45°, 22.5°, 0° with 16 different illumination conditions and neutral expression. From FERET database, face images of poses 60°, 40°, 15° and 0° are selected in neutral expression and illumination condition. For pre-processing, all images are cropped manually (such that eyes and mouth level are fixed), resized to 28×28 pixels and histogram equalization is performed on them.

As mentioned in Section 4, it is necessary to know the pose of a non-frontal face image in order to generate its virtual frontal view, because it is necessary to select the proper view dictionary. Table 1 and Table 2 show the accuracy of pose estimation algorithm for 5 poses (22.5° pose interval) and 3 poses (45° pose interval) of CMUPIE database. In each pose, 10 face images from 68 identities are randomly selected to construct the view dictionary. The remaining 58 images per pose are used for evaluation. The accuracies reported in these two tables are obtained by averaging several runs.

Both Tables 1 and 2 represent high accuracy (above 90%) in pose estimation, which is acceptable for many applications. The comparison of the results from these two tables implies that the accuracy of pose estimation increases when the difference between adjacent poses (pose interval) increases. This is because the dictionary atoms of different poses are more distinct and adjacent poses are more discriminable in large pose intervals. Although there might be methods in pose estimation with higher accuracies in small pose intervals, the proposed pose estimation method is simple, fast and accurate enough for the aim of virtual view generation and face recognition. Also, it is based on sparse representation which is the base for the other two steps of the proposed method, too.

The following experiment assesses the frontal view generation step of the proposed algorithm. Figure 8 shows the virtual frontal views generated from different methods for 22.5°, 45°, 90° poses. As the figure shows, compared to the generated frontal faces by LSRR or GLR methods, generated faces by the proposed method (MPSDL) are visually more similar to the ground-truth images. In fact, generated faces by GLR do not contain much details and generated faces for different identities are similar with artifacts for pose 90°. SRR generated faces have less artifacts and are visually acceptable, but are over-smoothed and details are lost. The LSRR method is similar to SRR, but it performs locally on small patches of an image, so its generated faces have less artifact but are locally smoothed. It requires many overlapped patches to generate detailed images. In contrast, the proposed MPSDL method can generate visually acceptable frontal faces which are similar to ground-truth images and have enough details to be used in recognizing identities.

For evaluating different view generation methods, a 10-fold cross validation strategy is used on CMU-PIE database. In each fold, 61 identities in 16 illuminations are selected for dictionary learning and the remaining 7 identities in 16 different illuminations are used for evaluation. Table 3 shows the Mean Square Error (MSE) between generated frontal views and ground-truths in three poses. The reported results in this table are based on 16 different illuminations of each pose. Chosen values for different parameters are mentioned in caption of the table where σ(B) is the standard deviation of matrix B. As can be seen from Table 3, for large pose variations (45° and 90°), the MSE between generated virtual frontal views and the ground-truths is smaller in the proposed method compared to GLR and SRR methods. The reason is that, compared to GLR, the proposed method has no assumption on linear transformation between different poses which is not correct for large pose angles. Also, compared to SRR, the proposed method is based on supervised dictionary learning which uses the discrimination in data and is expected to generate faces with more details. Since the proposed method is a holistic approach, LSRR and LLR methods are not included in this comparison, because of their locality manner. Obtained results in Table 3 clearly show that the proposed method can generate accurate virtual views even in large pose variations which can be considered as a desired property of the proposed method. As expected, MSE of different methods is more similar in small pose angles where transformation between poses is nearly linear.

It should be noted that the proposed method aims to generate discriminative virtual frontal views and it does not concentrate on visually good generated faces. However, the results in Table 3 depict that based on MSE, the proposed method can generate more accurate virtual frontal views in large pose angles, compared to other methods.

In this section, we evaluate the performance of the proposed multi-pose face recognition method. Virtual view generation can be considered as a preprocessing that is independent of the classification step. Therefore, face recognition based on generated frontal views can be done with various kind of classifiers. Since the SRC method has been successful in frontal face recognition task (Wright et al., 2009), in this paper, SRC is used as a classifier for the generated virtual views. The performance of the proposed method is compared to GLR, SRR and LSRR which are all based on virtual view generation. 10-fold cross validation is used to evaluate the performance of each method on CMU-PIE and FERET databases. For CMU-PIE database, each fold contains 7 identities in 5 poses and 16 different illumination conditions. For FERET database, each fold contains 20 identities in 4 poses and neutral illumination and expression conditions. Using this setting, recognition accuracy of different methods on CMU-PIE and FERET databases are shown in Table 4 and Table 5, respectively. In these tables, it is assumed that the pose angle of test images is known and the proposed pose estimation in Algorithm 1 is not used. In both tables, dictionary size parameter is adjusted to 200 for SRR, LSRR and MPSDL methods. Also, for LSRR, images have been partitioned into 16 patches.

Discussing the results reported in Tables 4 and 5, the recognition accuracy using raw images without virtual view generation will decrease rapidly when the pose angle increases. The GLR method performs better compared to raw images and improves the accuracy, however, its performance is not satisfying for large pose angles because of using linear assumption in virtual view generation. The SRR generally performs better than GLR for large pose angles. Using local patches, LSRR improves the accuracy of SRR, but it seems that both methods suffer from unsupervised dictionary learning. The proposed MPSDL method outperforms other methods and is more robust in large pose variations.

In order to investigate the impact of automatic pose estimation in step 1 on recognition accuracy, Table 6 shows the results of the proposed multi-pose face recognition on FERET database while using Algorithm 1 for pose estimation. As expected, the results in this table are very close to those of Table 5 which confirms the perfect performance of Algorithm 1 for pose estimation. Therefore, the effect of pose estimation error on recognition accuracy can be ignored.

The effect of dictionary size in recognition accuracy has been investigated through experiments done with different dictionary sizes. It should be noted that for raw images and the GLR method that are not based on dictionary learning, dictionary size points to the number of samples in a training set. Table 7 and Fig. 9 show the recognition accuracy of different methods under different dictionary sizes for 22.5° pose of CMU-PIE database. For small dictionary sizes, LSRR method outperforms others because of using small patches.

Increasing the dictionary size, face recognition accuracy increases in all methods of Table 7 (except for GLR), but as can be seen, performance of MPSDL increases more rapidly. For large dictionary sizes, accuracy of MPSDL overtakes other methods which is because of the use of supervised dictionary learning. Compared to other methods, GLR method shows weak performance for large dictionary sizes, which is because of overfitting of regression based methods in large number of training samples. However, it should be noted that the use of huge databases and big dictionaries might have some computational problems due to the use of huge memory size and very long processing time. One might cope with these problems using high-performance hardware devices and a quantization or clustering method prior to dictionary learning.

Although the proposed method has better performance compared to similar methods, it cannot overtake the state-of-the-art methods which are not based on sparse representation and 2D virtual view generation. For instance, Table 8 shows the comparison between the proposed method and some well-known and state-of-the-art methods in multi-pose face recognition on CMU-PIE database. As can be inferred from this table, 3D reconstruction based methods such as 3DMM (Blanz and Vetter, 2003) and Probabilistic Geometry Assisted FR (Liu and Chen, 2005) can recognize face images with higher accuracy, because they utilize the 3D model of face by aligning 2D images with either a general or an identity specific 3D model which is computationally expensive. TFA (Prince et al., 2008) is a 2D method which benefits from the latent variable view point for face recognition, but it requires two face images (one frontal and one non-frontal) in its gallery for recognition while the proposed method only requires one frontal face image. Therefore, it can be concluded that the recognition accuracy of the proposed method for multi-pose face recognition is persuasive from a computational point of view and when there is only one frontal image of each person in hand. However, based on the idea and obtained results in Zhang et al. (2015), Zhao et al. (2016), one can extend the proposed method by using mixed norm l(p,q) instead of l1 norm and hopefully decrease the frontal face generation error while increasing the face recogntion accuracy.