Recently, a fast growth of emotion recognition research has been observed in various types of communication such as text (Shivhare and Khethawat, 2012; Calvo and Kim, 2013; Ramalingam et al., 2018), speech (Tamulevičius et al., 2017, 2019; Sailunaz et al., 2018), body gestures (Stathopoulou and Tsihrintzis, 2011; Metcalfe et al., 2019), and facial expressions (Revina and Emmanuel, 2018; Ko, 2018; Shao and Qian, 2019; Sharma et al., 2019).

Facial expressions are one of the most important means of interpersonal communication, since a facial expression says a lot without speaking. Therefore, research on facial emotions has received much attention in recent decades in applications in the perceptual and cognitive sciences (Purificación and Pablo, 2019). Facial emotion recognition (FER) is widely used in distinct areas such as: neurology (Adolphs and Anderson, 2018; Metcalfe et al., 2019), clinical psychology (Su et al., 2017), artificial intelligence (Ranade et al., 2018), intelligent security (Wang and Fang, 2008), robotics manufacturing (Weiguo et al., 2004), behavioural sciences (Vorontsova and Labunskaya, 2020), multimedia (Mariappan et al., 2012), educational software (Ferdig and Mishra, 2004; Filella et al., 2016), etc.

In the literature, the common approach to facial emotion recognition consists of these steps: image pre-processing (noise reduction, normalization), face detection, facial feature extraction, and facial expression classification (recognition). Numerous techniques have been made for FER by using different methods in these steps (Bhardwaj and Dixit, 2016; Deshmukh et al., 2017; Ko, 2018; Revina and Emmanuel, 2018; Shao and Qian, 2019; Sharma et al., 2019). In the literature, recognition accuracy of this approach varies from approximately 48% to 98% (Deshmukh et al., 2017; Revina and Emmanuel, 2018; Shao and Qian, 2019; Nonis et al., 2019; Sharma et al., 2019). However, the common approach has some drawbacks (Shao and Qian, 2019): a) recognition accuracy is highly dependent on the methods used and the data set analysed; b) methods are often difficult, because of many unknown parameters and/or long computation time.

Recently, deep-learning-based algorithms have been employed for feature extraction, classification, and recognition tasks. The convolutional neural networks and the recurrent neural networks have been applied in many studies including object recognition, face recognition, and facial emotion recognition as well. However, deep-learning-based techniques are available with big data (Nonis et al., 2019). A brief review of conventional FER approaches as well as deep-learning-based FER methods is presented in Ko (2018). It is shown that the average recognition accuracy of six conventional FER approaches is equal to 63.2% and the average recognition accuracy of six deep-learning-based FER approaches is 72.65%, i.e. deep-learning based approaches outperform conventional approaches. In Gan et al. (2019), a novel FER framework via convolutional neural networks with soft labels that associate multiple emotions to each expression image is proposed. Investigations are made on the FER-2013 (35 887 face images) (Goodfellow et al., 2013), SFEW (1766 images) (Dhall et al., 2015) and RAF (15 339 images) (Li et al., 2017) databases, and the proposed method achieves accuracy of 73.73%, 55.73% and 86.31%, respectively.

In this paper, we focus on emotion recognition by facial expression. We have developed an approach, based on the two-dimensional model of emotions as well as using the kriging predictor of Fractional Brownian Vector Field (Motion) (FBVF). The classification problem, related to the recognition of facial emotions, is formulated and solved. The relationship of different emotions is estimated by expert psychologists by putting different emotions as the points on the plane. The kriging predictor allows us to get an estimate of a new picture emotion on the plane. Then, we determine which emotion, identified by psychologists, is the closest one. Seven emotions (Joy, Sadness, Surprise, Disgust, Anger, Fear, and Neutral) have been chosen for recognition.

The advantage of our method is that it is focused on small data sets. In the literature, seven basic emotions (e.g. Joy, Sadness, Surprise, Disgust, Anger, Fear, and Neutral) are usually used. However, sometimes specific emotions are measured. In this case, classical databases with basic emotions cannot be used for training of classifier. If we have little data for the study and cannot adapt other databases, then methods such as CNN will not give good accuracy with a small data set. This is an advantage of the kriging method. Our approach can be easily extended to other emotions.

Emotions can be expressed in a variety of ways, such as facial expressions and gestures, speech, and written text. There are two models to recognize emotions: the categorical model and the dimensional one. In the first model, emotions are described with a discrete number of classes, affective adjectives, and, in the second model, emotions are characterized by several perpendicular axes, i.e. by defining where they lie in a two, three or higher dimensional space (Grekow, 2018). The review of these models is made in Sreeja and Mahalakshmi (2017), Grekow (2018).

There are many attempts in the literature to visualize similarities of emotions. This allows them to be compared not only qualitatively but also quantitatively. Such visualizations, namely the quantitative correspondence of emotions to points on the 2D plane, are reviewed below. We rely on this in the proposed new method of recognizing and classifying facial emotions.

Recently, Fractional Brownian Vector Field (Motion) (FBVF) has been very popular among mathematicians and physicists (Yancong and Ruidong, 2011; Tan et al., 2015). The created model for FER is based on modelling valence and arousal dimensions in Russell’s model by the two-dimensional FBVF. Hereinafter, these dimensions are called coordinates as well.

Stochastic model of facial emotions on pictures should incorporate uncertainty about quantities in unobserved points and to quantify the uncertainty associated with the kriging estimator. Namely, the emotion at each facial picture is considered as a realization of FBVF Z(X,ω) , Z:Rn⊗Ω→R2 ,

which for every point in the variables space X∈Rn is a measurable function of random event ω∈(Ω,Σ,P) in some probability space (Pozniak et al., 2019). As it is unknown which of all function variables will be preponderant, consider them as equivalent, thus, calculate a distance between measurement points, which now is symmetric with respect to the miscellaneous variables. Usually it is assumed the FBVF has a constant mean vector and covariance matrix at each point: μ=(μ1,μ2)andβ=β11β12β21β22,β>0. 0.\]]]>

Thus, assume, the set X={X1,…,XN} of observed mutually disjoint vectors Xi∈Rn , 1⩽i⩽N , N>1 1$]]>, n⩾1 , where each vector represents one facial picture, is fixed, and data of measurement Y=(Y1,Y2,…,YN)T of the response vector surface, representing the emotion dimensions, at points of X is known, Yi=Z(Xi,ω) . Hence, the matrix of fractional Euclidean distances is computed as well: A=[|Xi−Xj|d]i,j=1N.

Degree d is a perfect parameter of FBVF as well, which can be estimated according to observation data. The maximal likelihood estimate dˆ ensuring asymptotically efficient and unbiased estimator can be estimated by minimization of logarithmic likelihood function: (1) dˆ=arg min0⩽d⩽1(ln(det(1N((YTA−1EETA−1YETA−1E−YTA−1Y)))2+ln|det(A)|N).

Novelty of our method is as follows: 1) We evaluate the Hurst parameter d by the maximum likelihood method; 2) We use a posteriori expectations and covariance matrix for kriging prediction of emotion model dimensions (coordinates); 3) We apply kriging predictor to FER in pictures.

Assume one has to predict the value of response vector surface Z at some point X∈Rn . Kriging gives us a way of anticipating, with some probability, a result associated with values of the parameters that have never been met before, or have been lost, to “store” the existing information (the experimental measurements), and propagate it to any situation where no measurement has been made. According to gentle introduction to kriging (Jones, 2001) and (Pozniak et al., 2019), it is defined by the kriging predictor which is defined as the conditional mean of FBVF: (2) Z(X)=YTA−1(a+E(1−ETA−1a)ETA−1E), where a is a distance vector, the elements of which are fractional Euclidean distances between a new (testing) data point and all the training data points.

This prediction is stochastic, its uncertainty is described by the conditional variance: (3) β(X)=β(aTA−1a−(1−ETA−1a)2ETA−1E), where the likelihood estimate of covariance matrix is applied: (4) β=YTA−1EETA−1YETA−1E−YTA−1Y.

Regarding the kriging model, the resent novelty is the introduction of d≠1 that expanded the possibilities of the model. So far, only d=1 was known in Dzemyda (2001). It is proved in Pozniak and Sakalauskas (2017) that the kernel matrix and the associated covariance matrix is positively defined, when 0⩽d<1 for any number of features and sample size. From the continuity of the likelihood function it follows that when there are more features (such as pixels) than the sample size (number of pictures), the covariance matrix can be positively defined when d>1 1$]]>, as well.

In this paper, the kriging predictor has been employed for emotion recognition from facial expression and explored experimentally because the kriging predictor performs simple calculations and has only one unknown parameter d, as well as because this method works very well with small data sets.

Warsaw set of emotional facial expression pictures (WSEFEP) (Olszanowski et al., 2015) has been used in the experiments. This set contains 210 high-quality pictures (photos) of 30 individuals (14 men and 16 women). They display six basic emotions (Joy, Sadness, Surprise, Disgust, Anger, Fear) and Neutral display. Examples of each basic emotion displayed by one woman are shown in Fig. 11.

The original size of these pictures was 1725×1168 pixels. In order to avoid the redundant information (background, hair, clothes, etc.), pictures were cropped and resized to 505×632 pixels (Fig. 12). Brows, eyes, nose, lips, cheeks, jaws, and chin are the key features that describe an emotional facial expression in obtained pictures.

Each picture has been digitized, i.e. a data point consists of colour parameters of pixels, and, therefore, it is of very large dimensionality. The number of pictures (data points) is N=210 . The images have 505×632 colour pixels (RGB), therefore their dimensionality is n=957480 .

Before presenting the kriging algorithm, some mathematical notations are introduced below. Suppose that the analysed data set X={X1,…,XN} consists of N n-dimensional points Xi=(xi1,…,xin) , i=1,N‾ ( Xi∈Rn ). The data point Xi corresponds to the ith picture in the picture set. Seven emotions (Joy, Sadness, Surprise, Disgust, Anger, Fear, and Neutral) are displayed in these pictures. For the sake of simplicity, the neutral state is attributed to the emotion as well. In this paper, for short, an emotion, identified from the facial expression shown in a particular picture, is called a picture emotion.

Since the two-dimensional circumplex space model of emotions (Fig. 10) is used for facial emotion recognition in the investigations, every emotion is represented as a point that has two coordinates: valence and arousal. The coordinates of the seven basic emotions (values of valence and arousal) are taken from Gobron et al. (2010) and are given in Paltoglou and Thelwall (2013). These coordinates are presented in Table 1.

As a picture emotion is known in advance, each data point Xi is related to an emotion point Yi=(yi1,yi2) that describes the ith picture emotion. Seven different combinations of ( yi1,yi2 ) are obtained (Table 1). In other words, yi1 and yi2 mean the valence and arousal coordinates, respectively, of the ith picture emotion in the two-dimensional circumplex emotion space (Fig. 10). Then, for the whole data set X , two column vectors y1 and y2 , the size of which is [N×1] , are comprised. The column vector y1 consists of the valence coordinates of the emotion points Yi , i=1,N‾ , and the column vector y2 consists of the arousal coordinates of these points, i.e. y1=(y11,y21,…,yN1)T and y2=(y12,y22,…,yN2)T .

The kriging predictor algorithm is as follows: 1.

The Euclidean distance matrix D between all the data points Xi , i=1,N‾ (from training data set) is calculated.

Here, A−1 is the inverse matrix of A, E is a unit column vector of size [N×1] , and a [N×1] is a distance vector, the elements of which are fractional Euclidean distances between a new (testing) data point and all the training data points. A new (testing) data point corresponds to a new picture whose emotion is being predicted. The training data points describe pictures whose emotions are known in advance. The meaning of y1 and y2 are described above. Outputs z1 and z2 correspond to the first and the second prediction parameter, respectively. In regard to the emotion model, employed in this research (Fig. 10), values of z1 and z2 mean the first (valence) and the second (arousal) coordinates, respectively, of the predicted emotion of a testing picture in the two-dimensional circumplex space.

The kriging predictor algorithm has only one unknown parameter d. The first investigation is performed seeking to find the optimal value of d. At first, the maximum likelihood (ML) function of picture emotion features y1 and y2 is determined as follows: (6) f=ln(|C|)2+ln(‖A‖)N, where ‖A‖ is an absolute value of the matrix A determinant, and |C| is a determinant of a posteriori covariance symmetric matrix C=(c11c12c21c22) , elements of which are calculated as follows: (7) c11=1N((y1TA−1E)2ETA−1E−y1TA−1y1),c22=1N((y2TA−1E)2ETA−1E−y2TA−1y2),c12=1N((y1TA−1E)(y2TA−1E)ETA−1E−y1TA−1y2),c21=c12.

In the next step, values of the ML function f are calculated for various values of the parameter d, i.e. d∈[0.01;1.05] . As a result, the dependence of the ML function f on the parameter d is obtained (Fig. 13). Figure 13 shows that this function is concave upward and has one local minimum as d=0.83 for the considered example.

The first investigation is pursued in order to recognize an emotion of a particular picture and evaluate the result obtained, as well as to verify that the optimal value dˆ=0.83 has been assessed properly.

In fact, we have a problem of classification into seven classes. Let the analysed picture data set X of size N be divided into two groups: testing and training data so that the testing data consist of only one picture and the training data are comprised of the remaining ones. In this way, N=210 experiments have been done. In the ith experiment, the ith picture emotion ( i=1,N‾ ) is identified. A classifier training leads to a kriging predictor training. According to formula (5), two coordinates ( z1 (valence) and z2 (arousal)) of this picture emotion are predicted by kriging and this picture emotion is mapped as a new point in the two-dimensional circumplex space. Then, a classification of the ith picture emotion is made. The task is to find out which of the seven basic emotions (Table 1) is the nearest one to the ith picture emotion, mapped in the emotion model (Fig. 10, Fig. 14). For this purpose, a measure of proximity, based on the Euclidean distances, is used. These distances are calculated between the mapped picture emotion and all the basic emotions (Table 1). The emotion that has the smallest distance to the analysed picture emotion is supposed to be the most suitable to identify the picture emotion. As a result, we get an emotion class to which the testing ith picture emotion belongs.

The efficiency of classifier will be estimated after such a run through all N experiments with picking different ith pictures for testing (N runs). Since the true picture emotions are known in advance, it is possible to find out how many picture emotions from the whole picture set ( N=210 ) are classified (recognized) successfully. Classification accuracy (CA) is calculated as the ratio of the number of correctly classified picture emotions to the total number of pictures as follows: (8) CA=the number of correctly classified picture emotionsthe total number of pictures100%.

Figure 15 illustrates the dependence of the picture emotion classification accuracy (CA) (%) on the parameter d, as d∈[0.1;1.05] . It is obvious from this figure that the best accuracy, i.e. CA∈[49%;50%] , is obtained as d∈[0.68;0.92] . When the optimal value of the parameter d is chosen, i.e. dˆ=0.83 , the classification accuracy is 50%. Since the best classification results are obtained as d∈[0.68;0.92] and the optimal value of the parameter d belongs to this range as well, i.e. dˆ=0.83∈[0.68;0.92] , it means that the optimal value dˆ=0.83 has been established properly by the ML method.

Figure 16 shows the mapping of predicted coordinates (valence and arousal) of all the 210 picture emotions in the two-dimensional circumplex space. It is obvious that Joy is predicted most precisely. However, the remaining emotions overlap quite strongly.

For deeper analysis of this classification, a confusion matrix of the seven basic emotions is given in Table 2. The highest true positive rates were observed for Joy (80%), Neutral (76.7%), and Disgust (60%). The highest false positive rates (the numbers are written in red) were observed for Anger (56.7% of pictures with Anger emotion were classified as Disgust), Fear (36.7% as Surprise), Sadness (36.7% as Neutral), and Surprise (33.3% as Fear).

The second investigation is similar to the first one because the ith picture emotion ( i=1,N‾ ) is identified, as well. However, in the 2nd investigation, differently from the 1st one, several basic emotions are combined into one group. At first, the three basic emotions, such as Fear, Anger, and Disgust, are combined into one group. It is reasonable to do this, because all the three emotions have the coordinates of negative valence and high arousal, i.e. they all are located in the second quarter of the analysed model of emotions (Fig. 14). In this case, we have a problem of classification into five classes: {Fear, Anger, Disgust}, {Surprise}, {Joy}, {Neutral}, and {Sadness}. Subsequently, the four emotions, i.e. Fear, Anger, Disgust, and Surprise, are grouped together. The decision to add the fourth emotion, i.e. Surprise, to the previous 3-emotion group is made because of the similarity of pictures with the Surprise and Fear emotions (see Fig. 12), as well as because Surprise and Fear are in a very near neighbourhood in the two-dimensional model of emotions (Fig. 14). For this reason, the picture emotion Surprise is very often classified as Fear and vice versa. So, we have a problem of classification into four classes: {Fear, Anger, Disgust, Surprise}, {Joy}, {Neutral}, and {Sadness}. Since the true picture emotions and emotion groups created are known in advance, the classification accuracy of the picture emotion set (size N) can be calculated. It is said that the picture emotion is identified rightly if the true picture emotion or emotion group (this picture emotion belongs to) is coincident with the identified one (emotion or group). Averaged values of the classification accuracy (%), when d∈[0.7;0.9] , are as follows: CA=50% , when emotions are not grouped, in the case of 3-emotion group, CA=64% , and, in the case of 4-emotion group, CA=76% . In this way, the classification accuracy is achieved rather well, i.e. 76% , when 4 emotions are grouped together.

Facial emotion recognition (FER) is an important topic in computer vision and artificial intelligence. We have developed the method for FER, based on the dimensional model of emotions as well as using the kriging predictor of Fractional Brownian Vector Field. The classification problem, related to the recognition of facial emotions, is formulated and solved. We use the knowledge of expert psychologists about the similarity of various emotions in the plane. The goal is to get an estimate of a new picture emotion on the plane by kriging and determine which emotion, identified by psychologists, is the closest one. Seven basic emotions (Joy, Sadness, Surprise, Disgust, Anger, Fear, and Neutral) have been chosen. The experimental exploration has shown that the best classification accuracy corresponds to the optimal value of Hurst parameter, estimated by the maximum likelihood method. The accuracy of classification into seven classes has been obtained approximately 50%, if we make a decision on the basis of the closest basic emotion. It has been ascertained that the kriging predictor is suitable for facial emotion recognition in the case of small sets of pictures. More sophisticated classification strategies may increase the accuracy, when grouping of the basic emotions is applied.