Automatic facial expression recognition is essential for many potential applications. Thus, having a clear overview on existing datasets that have been investigated within the framework of face ...expression recognition is of paramount importance in designing and evaluating effective solutions, notably for neural networks-based training. In this survey, we provide a review of more than eighty facial expression datasets, while taking into account both macro- and micro-expressions. The proposed study is mostly focused on spontaneous and in-the-wild datasets, given the common trend in the research is that of considering contexts where expressions are shown in a spontaneous way and in a real context. We have also provided instances of potential applications of the investigated datasets, while putting into evidence their pros and cons. The proposed survey can help researchers to have a better understanding of the characteristics of the existing datasets, thus facilitating the choice of the data that best suits the particular context of their application.
In this paper, we propose a novel solution for the problem of segmenting macro- and micro-expression frames (or retrieving the expression intervals) in video sequences, which is a prior step for many ...expression recognition algorithms. The proposed method exploits the non-rigid facial motion that occurs during facial expressions by capturing the optical strain corresponding to the elastic deformation of facial skin tissue. The method is capable of spotting both macro-expressions which are typically associated with expressed emotions and rapid micro- expressions which are typically associated with semi-suppressed macro-expressions. We test our algorithm on several datasets, including a newly released hour-long video with two subjects recorded in a natural setting that includes spontaneous facial expressions. We also report results on a dataset that contains 75 feigned macro-expressions and 37 feigned micro-expressions. We achieve over a 75% true positive rate with a 1% false positive rate for macro-expressions, and a nearly 80% true positive rate for spotting micro-expressions with a .3% false positive rate.
•Temporally segments macro- and micro-facial expressions from video•Does not rely on trained model of particular expression(s)•Measures the strain (deformation) impacted on facial skin tissue•The method successfully detects both spontaneous and feigned expressions.•The method works at several pixel resolutions.
Micro-expressions can reflect an individual’s subjective emotions and true mental state and are widely used in the fields of mental health, justice, law enforcement, intelligence, and security. ...However, the current approach based on image and expert assessment-based micro-expression recognition technology has limitations such as limited application scenarios and time consumption. Therefore, to overcome these limitations, this study is the first to explore the brain mechanisms of micro-expressions and their differences from macro-expressions from a neuroscientific perspective. This can be a foundation for micro-expression recognition based on EEG signals. We designed a real-time supervision and emotional expression suppression (SEES) experimental paradigm to synchronously collect facial expressions and electroencephalograms. Electroencephalogram signals were analyzed at the scalp and source levels to determine the temporal and spatial neural patterns of micro- and macro-expressions. We found that micro-expressions were more strongly activated in the premotor cortex, supplementary motor cortex, and middle frontal gyrus in frontal regions under positive emotions than macro-expressions. Under negative emotions, micro-expressions were more weakly activated in the somatosensory cortex and corneal gyrus regions than macro-expressions. The activation of the right temporoparietal junction (rTPJ) was stronger in micro-expressions under positive than negative emotions. The reason for this difference is that the pathways of facial control are different; the production of micro-expressions under positive emotion is dependent on the control of the face, while micro-expressions under negative emotions are more dependent on the intensity of the emotion.
With the growth of popularity of facial micro-expressions in recent years, the demand for long videos with micro- and macro-expressions remains high. Extended from SAMM, a micro-expressions dataset ...released in 2016, this paper presents SAMM Long Videos dataset for spontaneous micro- and macro-expressions recognition and spotting. SAMM Long Videos dataset consists of 147 long videos with 343 macro-expressions and 159 micro-expressions. The dataset is FACS-coded with detailed Action Units (AUs). We compare our dataset with Chinese Academy of Sciences Macro-Expressions and Micro-Expressions (CAS(ME) 2 ) dataset, which is the only available fully annotated dataset with micro- and macro-expressions. Furthermore, we preprocess the long videos using OpenFace, which includes face alignment and detection of facial AUs. We conduct facial expression spotting using this dataset and compare it with the baseline of MEGC III. Our spotting method outperformed the baseline result with F1-score of 0.3299.
Micro-expressions are the subtle and rapid movements of human facial expressions that could reveal a person's true emotions, including emotions that people attempt to suppress, hide, or restrain. ...Many recent papers have researched facial expression recognition systems in video sequences using GRU models. However, they haven't found a good relevance of micro-expression (ME) in detecting deceptive behaviors. In order to improve and contribute to the development of the system, we propose a micro-expression lie detection system with GRU's hyperparameter optimization and explore its accuracy. FER-2013 is used for expression recognition learning and the dataset containing video clips of courtroom trials is used for deception detection learning. Several normalization techniques are done in the process. The CNN model with eight convolutional layers and three fully linked layers is used to train the facial expression recognition system. To improve its accuracy, multiple GRU parameter settings are employed. We used the model on the test dataset after training it. The outcome shows that it was 92.31% accurate. The confusion matrix predicts 12 out of 13 outcomes, with 100% accuracy on the deceptive class and 85% accuracy on the truthful class.
This paper presents the reproduction of two studies focused on the perception of micro and macro expressions of Virtual Humans (VHs) generated by Computer Graphics (CG), first described in 2014 and ...replicated in 2021. The 2014 study referred to a VH realistic, whereas, in 2021, it referred to a VH cartoon. In our work, we replicate the study by using a realistic CG character. Our main goals are to compare the perceptions of micro and macro expressions between levels of realism (2021 cartoon versus 2023 realistic) and between realistic characters in different periods (i.e., 2014 versus 2023). In one of our results, people more easily recognized micro expressions in realistic VHs than in a cartoon VH. In another result, we show that the participants’ perception was similar for both micro and macro expressions in 2014 and 2023.
Deception is a very common phenomenon and its detection can be beneficial to our daily lives. Compared with other deception cues, micro-expression has shown great potential as a promising cue for ...deception detection. The spotting and recognition of micro-expression from long videos may significantly aid both law enforcement officers and researchers. However, database that contains both micro-expression and macro-expression in long videos is still not publicly available. To facilitate development in this field, we present a new database, Chinese Academy of Sciences Macro-Expressions and Micro-Expressions (CAS(ME)Formula Omitted), which provides both macro-expressions and micro-expressions in two parts (A and B). Part A contains 87 long videos that contain spontaneous macro-expressions and micro-expressions. Part B includes 300 cropped spontaneous macro-expression samples and 57 micro-expression samples. The emotion labels are based on a combination of action units (AUs), self-reported emotion for every facial movement, and the emotion types of emotion-evoking videos. Local Binary Pattern (LBP) was employed for the spotting and recognition of macro-expressions and micro-expressions and the results were reported as a baseline evaluation. The CAS(ME)Formula Omitted database offers both long videos and cropped expression samples, which may aid researchers in developing efficient algorithms for the spotting and recognition of macro-expressions and micro-expressions.
Deception is a very common phenomenon and its detection can be beneficial to our daily lives. Compared with other deception cues, micro-expression has shown great potential as a promising cue for ...deception detection. The spotting and recognition of micro-expression from long videos may significantly aid both law enforcement officers and researchers. However, database that contains both micro-expression and macro-expression in long videos is still not publicly available. To facilitate development in this field, we present a new database, Chinese Academy of Sciences Macro-Expressions and Micro-Expressions (CAS(ME)<inline-formula><tex-math notation="LaTeX">^2</tex-math> <inline-graphic xlink:href="fu-ieq2-2654440.gif"/> </inline-formula>), which provides both macro-expressions and micro-expressions in two parts (A and B). Part A contains 87 long videos that contain spontaneous macro-expressions and micro-expressions. Part B includes 300 cropped spontaneous macro-expression samples and 57 micro-expression samples. The emotion labels are based on a combination of action units (AUs), self-reported emotion for every facial movement, and the emotion types of emotion-evoking videos. Local Binary Pattern (LBP) was employed for the spotting and recognition of macro-expressions and micro-expressions and the results were reported as a baseline evaluation. The CAS(ME)<inline-formula><tex-math notation="LaTeX">^2</tex-math> <inline-graphic xlink:href="fu-ieq3-2654440.gif"/> </inline-formula> database offers both long videos and cropped expression samples, which may aid researchers in developing efficient algorithms for the spotting and recognition of macro-expressions and micro-expressions.
•For the first time, AUs and optical flow features are combined to spot either macro- or micro- expression intervals.•The proposal can eliminate the influence of facial image change caused by noises, ...such as body or head movement.•The proposed Concat-CNN model can learn both the inner features of a single frame and the correlation between frames.•The re-labeling method considers the overall change process of a specific expression and improves the detection performance.•The proposal shows remarkable improvement in the F1 scores on datasets, such as the CAS(ME)2-cropped and the SAMM-LV.
Display omitted
This paper is an extension of our previously published ACM Multimedia 2022 paper, which was ranked 3rd in the macro-expressions (MaEs) and micro-expressions (MEs) spotting task of the FME challenge 2021. In our earlier work, a deep learning framework based on facial action units (AUs) was proposed to emphasize both local and global features to deal with the MaEs and MEs spotting tasks. In this paper, an advanced Concat-CNN model is proposed to not only utilize facial action units (AU) features, which our previous work proved were more effective in detecting MaEs, but also to fuse the optical flow features to improve the detection performance of MEs. The advanced Concat-CNN proposed in this paper not only considers the intra-features correlation of a single frame but also the inter-features correlation between frames. Further, we devise a new adaptive re-labeling method by labeling the emotional frames with distinctive scores. This method takes into account the dynamic changes in expressions to further improve the overall detection performance. Compared with our earlier work and several existing works, the newly proposed deep learning pipeline is able to achieve a better performance in terms of the overall F1-scores: 0.2623 on CAS(ME)2, 0.2839 on CAS(ME)2-cropped, and 0.3241 on SAMM-LV, respectively.
•The STCEAN model considers the changes of spatial features in the temporal dimension.•The MAS of two heads is focus on different emotional dimensions attention weight.•The STCEAN model uses focal ...loss function to reduce sample imbalance.•Leave-Half-Subject-Out (LHSO) cross-validation method to reduce trained time.
Emotional detection based on facial micro-expressions is essential in high-risk tasks such as criminal investigation or lie detection. However, micro-expressions often occur in high-risk tasks when people often use facial expressions to conceal their actual emotional states. Therefore, spotting macro- and micro-expression intervals in long video sequences has become hot research. Considering the difference in duration and facial muscle movement intensity between macro- and micro-expression, we propose a novel Spatio-temporal Convolutional Emotional Attention Network (STCEAN) for spotting macro- and micro-expression intervals in long video sequences. The spatial features of each frame in the video sequence are extracted through the convolution neural network. Then the emotional self-attention model is used to analyze the temporal weights of spatial features in different emotional dimensions. The emotional weights in the temporal dimension are filtered for spotting macro- and micro-expressions intervals. Finally, the STCEAN model is jointly optimized by the dual emotional focal loss of macro- and micro-expression to solve the problem of sample unbalance. The experimental results on the CAS(ME)2 and SAMM-LV datasets show that the STCEAN model achieves competitive results in the Facial Micro-Expression Challenge 2021.
PDF
