Emotion recognition is the ability to precisely infer human emotions from numerous sources and modalities using questionnaires, physical signals, and physiological signals. Recently, emotion ...recognition has gained attention because of its diverse application areas, like affective computing, healthcare, human–robot interactions, and market research. This paper provides a comprehensive and systematic review of emotion recognition techniques of the current decade. The paper includes emotion recognition using physical and physiological signals. Physical signals involve speech and facial expression, while physiological signals include electroencephalogram, electrocardiogram, galvanic skin response, and eye tracking. The paper provides an introduction to various emotion models, stimuli used for emotion elicitation, and the background of existing automated emotion recognition systems. This paper covers comprehensive searching and scanning of well-known datasets followed by design criteria for review. After a thorough analysis and discussion, we selected 142 journal articles using PRISMA guidelines. The review provides a detailed analysis of existing studies and available datasets of emotion recognition. Our review analysis also presented potential challenges in the existing literature and directions for future research.
•A systematic review of emotion recognition comprising journal articles of the last decade culled from renowned databases.•Inclusion of physical and physiological signals used for emotion recognition using artificial intelligence.•A detailed examination of fine-grained emotion recognition research, including resources, approaches, and datasets.•Identifying research gaps and future research directions for emotion recognition.
Full text
Available for:
GEOZS, IJS, IMTLJ, KILJ, KISLJ, NLZOH, NUK, OILJ, PNG, SAZU, SBCE, SBJE, UILJ, UL, UM, UPCLJ, UPUK, ZAGLJ, ZRSKP
Most existing automatic kinship verification methods focus on learning the optimal distance metrics between family members. However, learning facial features and kinship features simultaneously may ...cause the proposed models to be too weak. In this work, we explore the possibility of bridging this gap by developing knowledge-based tensor models based on pre-trained multi-view models. We propose an effective knowledge-based tensor similarity extraction framework for automatic facial kinship verification using four pre-trained networks (i.e., VGG-Face, VGG-F, VGG-M, and VGG-S). Therefore, knowledge-based deep face and general features (such as identity, age, gender, ethnicity, expression, lighting, pose, contour, edges, corners, shape, etc.) were successfully fused by our tensor design to understand the kinship cue. Multiple effective representations are learned for kinship verification statements (children and parents) using a margin maximization learning scheme based on Tensor Cross-view Quadratic Exponential Discriminant Analysis. Through the exponential learning process, the large gap between distributions of the same family can be reduced to the maximum, while the small gap between distributions of different families is simultaneously increased. The WCCN metric successfully reduces the intra-class variability problem caused by deep features. The explanation of black-box models and the problems of ubiquitous face recognition are considered in our system. The extensive experiments on four challenging datasets show that our system performs very well compared to state-of-the-art approaches.
•We derive a novel hybrid knowledge-based multi-linear subspace based on deep multiview features for kinship verification.•Deep multi-view features are realized, fused and analyzed via a tensor design to separate the multi-factorial structure related to kinship verification of facial images.•A multi-linear cross-view design is proposed for optimal margin maximization and dimensionality reduction using the concept of exponential matrix.•Experiments demonstrate the robustness of our system using four facial kinship databases.
Full text
Available for:
GEOZS, IJS, IMTLJ, KILJ, KISLJ, NLZOH, NUK, OILJ, PNG, SAZU, SBCE, SBJE, UILJ, UL, UM, UPCLJ, UPUK, ZAGLJ, ZRSKP
Facial expression recognition (FER) is a significant task for the machines to understand the emotional changes in human beings. However, accurate hand-crafted features that are highly related to ...changes in expression are difficult to extract because of the influences of individual difference and variations in emotional intensity. Therefore, features that can accurately describe the changes in facial expressions are urgently required. Method: A weighted mixture deep neural network (WMDNN) is proposed to automatically extract the features that are effective for FER tasks. Several pre-processing approaches, such as face detection, rotation rectification, and data augmentation, are implemented to restrict the regions for FER. Two channels of facial images, including facial grayscale images and their corresponding local binary pattern (LBP) facial images, are processed by WMDNN. Expression-related features of facial grayscale images are extracted by fine-tuning a partial VGG16 network, the parameters of which are initialized using VGG16 model trained on ImageNet database. Features of LBP facial images are extracted by a shallow convolutional neural network (CNN) built based on DeepID. The outputs of both channels are fused in a weighted manner. The result of final recognition is calculated using softmax classification. Results: Experimental results indicate that the proposed algorithm can recognize six basic facial expressions (happiness, sadness, anger, disgust, fear, and surprise) with high accuracy. The average recognition accuracies for benchmarking data sets "CK+," "JAFFE," and "Oulu-CASIA" are 0.970, 0.922, and 0.923, respectively. Conclusions: The proposed FER method outperforms the state-of-the-art FER methods based on the hand-crafted features or deep networks using one channel. Compared with the deep networks that use multiple channels, our proposed network can achieve comparable performance with easier procedures. Fine-tuning is effective to FER tasks with a well pre-trained model if sufficient samples cannot be collected.
In response to the problem that facial features are likely to cause privacy leakage and identity theft of the user, this paper presents a method for generating De-identification facial images based ...on attention models and adversarial examples. The method has two training stages. In Stage 1, target classification networks are used as an attention module to extract Refined Features of a facial image to perform feature fusion and obtain the face feature fusion matrix. In Stage 2, the face feature fusion matrix is used as an initial data distribution of the generator in the adversarial generation network and adds pixel-level constraint loss functions and loss functions for cutting adversarial example perturbations. The purpose of this is for ensuring the generated private image of the face is usable and add attention modules to discriminator of generative adversarial network. Next, the method extracts the feature matrix of the de-identity face image and trains it to mimic the face feature fusion matrix to increase the migration of the adversarial response in the de-identity face image. Experimental results have shown that the performance of the proposed method is superior to or approach state-of-the-art methods in terms of image quality and robustness.
Full text
Available for:
GEOZS, IJS, IMTLJ, KILJ, KISLJ, NLZOH, NUK, OILJ, PNG, SAZU, SBCE, SBJE, UILJ, UL, UM, UPCLJ, UPUK, ZAGLJ, ZRSKP
Facial images are widely adopted for computer vision tasks such as face recognition or attribute classifications. Consequently, the adoption of mass real facial images leads to significant ...identification privacy leakage concerns. Meanwhile, the model classification results suffer unfair predictions towards features such as genders due to biased training data distributions. Although methods have been proposed to resolve the privacy and fairness issues separately, simultaneous protection methods are merely studied. In this study, for facial attributes classifications, we propose one unified framework with GAN models to generate synthetic images for privacy protections and contrastive learning based loss designs to enforce fairness protections simultaneously. Meanwhile, unlike other privacy or fairness protection methods, the proposed methods can maintain high data and model utilities. We evaluate our approaches with the high image resolution dataset CelebA-HD, and the results show our methods meet both privacy and fairness requirements.
Full text
Available for:
GEOZS, IJS, IMTLJ, KILJ, KISLJ, NLZOH, NUK, OILJ, PNG, SAZU, SBCE, SBJE, UILJ, UL, UM, UPCLJ, UPUK, ZAGLJ, ZRSKP
The objective of this study was to define a more practical and reliable alternative to manual temperament classification methods that rely on the behavioral responses of animals individually ...subjected to various tests. Specifically, this study evaluated the correlation between facial image information and temperament based on deep convolutional neural networks (DCNNs) to predict the temperament of lambs based on their facial images. In the first phase, the lambs were categorized as to their temperament based on data acquired from a behavioral test to establish a ground truth for the temperament of the lambs. This enabled us to train (70%), validate (20%), and test (10%) deep-learning models in the second phase based on facial images and the corresponding temperament labels derived from the behavioral test. The performance of a custom deep convolutional neural network (C-DCNN) was compared to that of pre-trained VGG19 and Xception models for image classification. The Xception model achieved a training accuracy of 81%, which indicated that it learned well the underlying patterns in the data; however, lower validation (0.75) and test (0.58) accuracies indicate that it overfit the training data and did not generalize well to new samples. The VGG19 model, produced lower training (0.59), validation (0.46), and test (0.34) accuracies, which indicated that it did not learn the underlying patterns in the data as well as the Xception model. Furthermore, its precision (0.47), recall (0.42), and F1 score (0.41) indicated that the model performed poorly in identifying the classes correctly. The C-DCNN produced a moderate accuracy of 60%, which indicated that the model was able to predict the temperament traits of lambs with an accuracy of 60%, which was better than random guessing (33% accuracy), and demonstrated the potential of this approach in assessing temperament. The C-DCNN precision (0.69), recall (0.61) and F1 score (0.63) indicated that it had a moderate ability to correctly identify positive cases; however, the small size of the original dataset remains a limitation of the study because it might have caused the suboptimal performance of the models. To validate this approach, further research is needed based on a larger and more diverse dataset. We will continue to investigate the potential of deep learning and computer vision to predict animal personality traits from facial images based on large, diverse datasets, which might lead to more efficient and objective methods for assessing animal temperament and improving animal welfare.
•FAMD and AHC analysis can be employed to objectively categorize the temperaments of lambs.•Facial images provide useful information about temperament traits of lambs.•Deep convolutional neural networks has the potential to correctly predict the temperament traits of the lambs.
Full text
Available for:
GEOZS, IJS, IMTLJ, KILJ, KISLJ, NLZOH, NUK, OILJ, PNG, SAZU, SBCE, SBJE, UILJ, UL, UM, UPCLJ, UPUK, ZAGLJ, ZRSKP
Research on face recognition has been evolving for decades. There are numerous approaches developed with highly desirable outcomes in constrained environments. In contrast, approaches to face ...recognition in an unconstrained environment where varied facial posing, occlusion, aging, and image quality still pose vast challenges. Thus, face recognition in the unconstrained environment still an unresolved problem. Many current techniques are not performed well when experimented in unconstrained databases. Additionally, most of the real-world application needs a good face recognition performance in the unconstrained environment. This paper presents a comprehensive process aimed to enhance the performance of face recognition in an unconstrained environment. This paper presents a face recognition system in an unconstrained environment. The fusion between Gabor filters and Maximum Response (MR) filters with Random Forest classifier is implemented in the proposed system. Gabor filters are a hybrid of Gabor magnitude filters and Oriented Gabor Phase Congruency (OGPC) filters. Gabor magnitude filters produce the magnitude response while the OGPC filters produce the phase response of Gabor filters. The MR filters contain the edge- and bar-anisotropic filter responses and isotropic filter responses. In the face features selection process, Monte Carlo Uninformative Variable Elimination Partial Least Squares Regression (MC-UVE-PLSR) is used to select the optimal face features in order to minimize the computational costs without compromising the accuracy of face recognition. Random Forests is used in the classification of the generated feature vectors. The algorithm performance is evaluated using two unconstrained facial image databases: Labelled Faces in the Wild (LFW) and Unconstrained Facial Images (UFI). The proposed technique used produces encouraging results in these evaluated databases in which it recorded face recognition rates that are comparable with other state-of-the-art algorithms.
PDF
