Remote-Url: https://www.pnas.org/content/111/15/E1454
Retrieved-at: 2021-08-04 10:48:59.786117+00:00

ResultsDatabase.If we are to build a database that can be successfully used in computer vision and machine learning experiments as well as cognitive science and neuroscience studies, data collection must adhere to strict protocols. Because little is known about compound emotions, our goal is to minimize effects due to lighting, pose, and subtleness of the expression. All other variables should, however, vary to guarantee proper analysis.Sample pictures for neutral and each of the six basic and 15 compound emotions are shown inFig. 1. Images were only accepted when the experimenter obtained fully recognizable expressions. Nonetheless, all images were subsequently evaluated by the research team. Subjects who had one or more incorrectly expressed emotions were discarded. The images of 230 subjects passed this evaluation (seeMaterials and Methods).Action Units Analysis.In their seminal work, Ekman and Friesen (12) defined a coding system that makes for a clear, compact representation of the muscle activation of a facial expression. Their Facial Action Coding System (FACS) is given by a set of action units (AUs). Each AU codes the fundamental actions of individual or groups of muscles typically seen while producing facial expressions of emotion. For example, AU 4 defines the contraction of two muscles resulting in the lowering of the eyebrows (with the emphasis being in the inner section). This AU is typically observed in expressions of sadness, fear, and anger (7).We FACS coded all of the images in our database. The consistently active AUs, present in more than 70% of the subjects in each of the emotion categories, are shown inTable 1. Typical intersubject variabilities are given in brackets; these correspond to AUs seen in some but not all individuals, with the percentages next to them representing the proportion of subjects that use this AU when expressing this emotion.Table 1.Prototypical AUs observed in each basic and compound emotion categoryAs expected, the AU analysis of the six basic emotions in our database is consistent with that given in ref.12. The only small difference is in some of the observed intersubject variability given in parentheses—i.e., AUs that some but not all subjects used when expressing one of the basic emotion categories; this is to be expected because our database incorporates a much larger set of subjects than the one in ref.12. Also, all of the subjects we have FACS coded showed their teeth when expressing happiness (AU 25), and this was not the case in ref.12. Moreover, only half of our subjects used AU 6 (cheek raiser) when expressing sadness, which suggests a small relevance of this AU as other studies have previously suggested (13⇓–15). Similarly, most of our subjects did not include AU 27 (mouth stretch) in fear, which seems to be active only when this expression is exaggerated.Table 1also lists the AUs for each of the compound emotion categories. Note that the AUs of the subordinate categories are used to form the compound category unless there is a conflict. For example, lip presser (AU 24) may be used to express disgust while lips part (AU 25) is used in joy. When producing the facial expression of happily disgusted, it is impossible to keep both. In this case, AU 24 is dropped.Fig. 2shows this and five other examples (further illustrated inTable 1). The underlined AUs of a compound emotion are present in both of their subordinate categories. An asterisk indicates the AU does not occur in either of the basic categories and is, hence, novel to the compound emotion. We did not find any such AU consistently used by most subjects; nevertheless, a few subjects did incorporate them, e.g., AU 25 (lips part) in sadly disgusted. Additional examples are given inFig. S1, where we include a figure with the subordinate relations for the nine remaining compound facial expressions of emotion.We note obvious and unexpected production similarities between some compound expressions. Not surprisingly, the prototypical AUs of hatred and appalled are the same, because they are both variations of angrily disgusted that can only be detected by the strength in the activation of their AUs. More interestingly, there is a noticeable difference in over half the subjects who use AU 7 (eyelid tightener) when expressing hate. Also interesting is the difference between the expression of these two categories and that of angrily disgusted, where AU 17 (chin raiser) is prototypical. These differences make the three facial expressions distinct from one another.The facial expression of sadly angry does not include any prototypical AU unique to anger, although its image seems to express anger quite clearly (Fig. 1K). Similarly, sadly fearful does not include any prototypical AU unique to sadness, but its image is distinct from that of fear (Fig. 1DandJ).