Journal of Nonverbal Behavior, 25, pp.167-186.

A Psychometric Evaluation of the Facial Action Coding System for Assessing Spontaneous Expression Michael A. Sayette, Jeffrey F. Cohn, Joan M. Wertz, Michael A. Perrott, and Dominic J. Parrott University of Pittsburgh Abstract The Facial Action Coding System (FACS) (Ekman & Friesen, 1978) is a comprehensive and widely used method of objectively describing facial activity. Little is known, however, about inter-observer reliability in coding the occurrence, intensity, and timing of individual FACS action units. The present study evaluated the reliability of these measures. Observational data came from three independent laboratory studies designed to elicit a wide range of spontaneous expressions of emotion. Emotion challenges included olfactory stimulation, social stress, and cues related to nicotine craving. Facial behavior was video-recorded and independently scored by two FACS-certified coders. Overall, we found good to excellent reliability for the occurrence, intensity, and timing of individual action units and for corresponding measures of more global emotion-specified combinations. Introduction After a long hiatus, research on emotion has emerged as an important topic of psychological inquiry (see Ekman, 1998; National Advisory Mental Health Council, 1995; Russell & Fernandez-Dols, 1997). Much credit for this renewed interest is attributable to the development beginning in the 1970s of observational coding systems to identify facial expressions thought to be associated with emotion. These include the Facial Action Scoring Technique (FAST: Ekman, Friesen, & Tomkins, 1971), A System for Identifying Affect Expressions by Holistic Judgment (AFFEX: Izard, Dougherty, & Hembree, 1983), EMFACS (Ekman & Friesen, 1982), Monadic Phases (Tronick, Als, & Brazelton, 1980), the Maximally Discriminative Facial Movement Coding System (MAX: Izard, 1979), and the Facial Action Coding System (FACS: Ekman & Friesen, 1978). Many of these systems enable researchers to make judgments about emotion state and to code facial expression using emotion labels. Such systems can be learned relatively quickly and have become highly influential (e.g., Campos, Barrett, Lamb, Goldsmith, & Stenberg, 1983). They have several limitations. Different systems give the same labels to different facial actions (e.g., Oster et al., 1992); they implicitly or explicitly assume that facial expression and emotion have an exact correspondence, which is problematic (see Camras, 1992; Fridlund, 1992; Russell, 1994), and paralinguistic and other nonverbal expressions, such as the brow flash, used as a greeting display in many parts of the world (e.g., Eibl-Eibesfeldt, 1989), are omitted. As a consequence, more descriptive methods of describing facial expression have become increasingly influential in emotion science (e.g., Ekman & Rosenberg, 1997). Chief among these are the


Kanade. While a proficiency test is required for certification as a FACS coder. Malatesta. 1988.g. It may be unwise to assume that good reliability for these measures in posed expression indicates good reliability in spontaneous expressions.. 1996). fails to differentiate between some anatomically distinct expressions (Oster et al. 1991) forensic science (Frank. Cohn. Fox & Davidson. on the assumption that FACS can be used reliably to code spontaneous emotional expressions.167-186.. while miscellaneous actions (e. 1997). 1998. often with the aim of producing expressions believed to represent emotion prototypes (e. & Sejnowski. Much of the research using FACS has involved posed expressions (Rosenberg. is more comprehensive than MAX (Oster et al. which can impede coding (e. In addition. Lien. 25. Levenson. European Conference on Facial Expression. and Meaning. FACS itself is purely descriptive.g. FACS has emerged as the criterion measure of facial behavior in multiple fields including computer vision (Bartlett. 1997. Hager. objective. which are decomposed into 30 action units (AUs) and 14 miscellaneous actions.g. action unit intensity). 1978). Ekman & Rosenberg. Combinations of FACS action units may be described with emotion labels if investigators choose to do so. jaw thrust) are ones for which the anatomic bases have not been established (Ekman & Friesen. Kanade. informed by the pioneering work of Hjortsjo (1969). & Nagel. and considers as separable expressions that are not anatomically distinct (Oster et al. & Ross. pp. Tian. computer graphics (Parke & Waters. AUs have a specified anatomic basis. however. 1988. Although studies of posed facial expression have generated a range of important findings. but this inferential step is extrinsic to FACS. 1997) studies of emotion. including velocity and smoothness of motion (Hager & Ekman. MAX (Izard. 1995). rigid head motion and face occlusion. 1992). Tesmna. 1992). 1989. The potential implications for such research rest. 1999).. To accommodate the greater range of head motion 2 . Rinn. & Cohn. 1989). Maximally Discriminative Facial Movement Coding System (MAX: Izard. 1999. & Tian. Because of its descriptive power. 1978). social (Frank & Ekman. unobtrusive. in press). 1979) and the Facial Action Coding System (FACS: Ekman & Friesen: 1978). Oster. Culver. 1992). Ekman. Ekman.. & Friesen. FACS (Ekman & Friesen. especially for the use of FACS with spontaneous facial expression..g. and reliable analysis of spontaneously generated expressions (see Ekman & Rosenberg. Measurement. 1997) and clinical (Part II. & Li. 1992). Cohn. & Shephard. relatively little information is available about the reliability of FACS coding for individual action units and related measures (e. 2000.Journal of Nonverbal Behavior. Hegley.. 1992. 1996). 2000). Participants are directed to voluntarily contract specific muscles. coders can manually code all possible facial displays..g. and developmental (Camras. Matias. 1983). Cohn. Kanade. neuroscience (Bruce & Young. Katsikitis & Pilowsky.. are more likely to occur during spontaneous expressions. the allure of FACS is that it also may provide an immediate. By comparison. Spontaneous expressions are believed to differ from voluntary ones in both their morphology and dynamics. 1979) provides less complete description of facial actions (e. Psychometric knowledge of FACS has not kept pace with the increasing and multiple uses of FACS coding in emotion science and related fields and in changes in FACS over time. Using FACS and viewing videotaped facial behavior in slow motion.

Ekman et al. Ellgring.. Consequently all FACS-certified coders presumably have achieved reliability. which affords a temporal resolution of either 1/30th or 1/25th of a second depending on the video standard (NTSC and PAL. Heller & Haynal. 1978). 1981) that correct for chance agreement between coders. Third. 1985). for instance. The intensity of facial movement is scored for five of the 30 action units (Ekman & Friesen. respectively. many published studies of spontaneous expression either fail to report FACS reliability (e. & Hecker 1990. investigators often use wider camera angles.. which reduces face size relative to the video frame and makes coding of subtle motion more difficult. Fleiss. for instance. 1994).g. 25. Friesen. The FACS certification test. agreement between observers) are relevant to the interpretation of substantive findings. 1997) or provide only incomplete information. Ekman.. Ekman. reported mean differences in AU onset between coders. higher resolution is available with special purpose video decks). found in studies of spontaneous expression. omits intensity differences altogether. Oster. spontaneous facial actions are believed to show greater symmetry in intensity of motion between left and right sides of the face (Hager & Ekman. Little is known. Russell. Friesen. Black. At least four types of inter-observer reliability (i. A second type of reliability is the temporal resolution of FACS coding. facial motion varies in degree as well as in type of AU shown. 1960. reliability at this more micro-level is needed.. 1992. and research reports often fail to evaluate measurement error in this regard. statistical power may be reduced by measurement error and negative findings misinterpreted. 1992). Brummett et al.. 1992. and Simons (1985). AUs typically are coded using stop-frame video. 1997). Banninger-Huber. however.. but see also Fridlund. Most studies report only reliability averaged across all AUs. 1994. Differences in the intensity of AUs are believed to relate to differences in intensity of subjective experience (Camras. Hypotheses about 3 . When specific AUs are the focus of hypothesis testing. Rosenberg & Ekman. If agreement statistics are not corrected for chance agreement. One is the reliability of individual AUs. & Bradshaw. even when intensity differences are focus of study (e. When hypotheses entail very small differences in latency or the duration of response (e. 1996. Even when reliability of individual AUs is assessed. reliability estimates will fail to generalize to populations in which the marginal distribution of AUs varies. precision of measurement is a critical concern. Cohn & Elmore. which may mask low reliability for specific ones (Ekman & Rosenberg.. 1992. It is important to test whether the reliability achieved for FACS certification is maintained in research studies. but not the distribution of errors as a function of variation in the time base. about whether coders can agree on intensity scoring.g. Miyake. 1985). It is unknown whether investigators can detect change in AUs on this time base. 1986.Journal of Nonverbal Behavior. pp. Nevertheless. Chesney.e. The FACS certification test requires coders to score videotapes of spontaneous expression with a high level of agreement with a group of reference coders. Information about reliability of measurement for measures of interest is important in planning new studies. In contrast to deliberate expressions. investigators typically fail to use information statistics such as kappa (Cohen.g. Camras et al. Campos. Otherwise. Failure to consider reliability of individual AUs may be of little concern when investigators analyze aggregate measures.167-186. 1998.

including facial expression (Sayette & Hufford. Odors often have been used to manipulate emotions (Aggleton & Mishkin. With few exceptions. This procedure has also been found to elicit potent emotional reactions across a range of response systems (Rohsenow. Reliability was assessed for the occurrence of single AUs. the precision with which AUs could be coded at temporal resolutions up to 1/30th of a second. Fourth. in many studies. Sayette & Wilson. one can more accurately estimate their reliability. Smith. investigators are interested in testing hypotheses about emotionspecified expressions. look at. hold.. This speech instruction has been found to increase negative affect across a range of measures (Levenson. Sher. The original decision to ignore them was made in part because of the difficulty in attaining reliability about very small motion. Under the traditional 3-point scoring. & Monti. surprise. 25. These studies provided suitable material for a rigorous test of the reliability of FACS for the occurrence of single AUs. Steele & Josephs. Grossman. Recent changes in the minimum criteria for scoring AUs (Friesen & Ekman. At issue is the comparability of studies that use the 3-point versus the 5-point approach to intensity scoring. pp. 1986. 1990-1991). smokers underwent a cue exposure manipulation.Journal of Nonverbal Behavior. 1992). To ensure a representative test-bed of spontaneous expression. The effect of scoring AUs at the trace level on FACS reliability is unknown. AU intensity. Sayette. and interobserver reliability for AU intensity and emotion-specified expressions. Breiner. in which they were asked to light. & Newlin. included both those thought to represent discrete emotions and more molar categories of positive and negative emotion. 1991. AUs at trace levels (AU intensity “a”) are now coded. Facial behavior was FACS coded from videotape by experienced pairs of coders certified in the use of FACS.167-186. Engen. In the second experiment. 1995). Emotion-specified expressions. which are considered to represent combinations of AUs. fear. Agreement between coders was quantified using kappa coefficients to control for chance levels of agreement. 4 . Niaura. 1988). 1992) make reliability assessment especially compelling. 1980.. intensity variation in AU intensity depend on whether coders can reliably agree on intensity.g. Participants were videotaped under typical emotion-eliciting laboratory conditions.g. which are defined in terms of specific combinations of AUs. anger. sadness. have produced facial expressions associated with affect (see Soussignan & Schaal. Abrams. & Wilson. 1992). “trace” levels of AUs were ignored. including facial expressive behavior (e. Childress. we included data from three experiments involving independent samples of subjects. Emotionspecified expressions include discrete emotions (e. reliability of intensity scoring of individual AUs is unknown. and emotion-specified aggregates at varying levels of temporal resolution. The reliability of emotion–specified expressions will of course depend on the constituent AUs. The first experiment induced emotion through the administration of odors that were preselected to elicit emotional responses. Newman. By assessing the reliability of these aggregates directly. but not smoke a cigarette. Under the new 5-point rules. joy. and disgust) and more molar distinctions between positive versus negative emotion. The present research evaluated inter-observer reliability of FACS in studies of spontaneous expression. and of most relevance to the present study. 1996). The third experiment induced an emotional response by instructing participants to present a self-disclosing speech pertaining to their physical appearance.

. reliability data were obtained by having two FACS raters independently code the expressions of a randomly selected subset of participants in an investigation of smoking craving (Sayette. pp. Martin. banana. see Sayette. Videotapes of the facial expressions were then coded using FACS. a vinegar solution. Facial behavior was recorded with the same equipment used in Experiment 1.g. [For details about the study’s methods.g. & Perrott. n = 12) smokers participating in a smoking cue exposure procedure. each ranging from 5-10 secs. In Experiment 3. Experiment 1. In each case. or adequate reliability. Inter-observer agreement was quantified with coefficient kappa. Coefficients of 0.75 or higher indicate excellent reliability (e. producing 114 segments. under review). To elicit a wide range of emotions.. when they were holding a lit cigarette. Method Overview The three laboratory experiments described here all involved the elicitation of spontaneous expressions. coefficients of 0.167-186. Odors included cooking extract oils (e.e. reliability data were obtained from 25 (13 male and 12 female) participants who were randomly selected from 169 participants who completed the experiment [for additional information about this study. Fleiss. smoke five or fewer cigarettes at least two days/week. In Experiment 2. a floral odor. 1981). Shiffman. peppermint. which is the proportion of agreement above what would be expected to occur by chance (Cohen.75 indicate good. in which they held and looked at a lit cigarette. and a neutral water odor [for additional information about this study. These segments were independently coded by two individuals who were certified in the use of FACS by the Social Interaction Laboratory at the University of California San Francisco (DP and JW. 58 participants (30 male and 28 female) were asked to rate the pleasantness of a series of eight odors. see also a similar study by Sayette and Hufford (1995)].Journal of Nonverbal Behavior. Six time periods were independently coded by two FACS coders. producing 116 segments. Martin. Fleiss. lemon) as well as a Vicks mixture. some of the smokers were deprived of nicotine for seven hours (n = 6) while others were nondeprived (n = 13). participants’ facial movements were videotaped using a single S-VHS camera and recorder that provided a frontal view of the subject’s chest and face. Perrott. each lasting about 12-14 seconds. coconut. see Sayette and Parrott (1999)]. MP and JW. These time periods included moments when participants first were presented with the cigarette. Wertz. Segments for coding were pre-selected for the likelihood of reflecting emotion in the face.. Experiments 2 and 3). 25. 1960. and Hufford (in 5 . Participants were videotaped using a Panasonic 450 Super-VHS recorder and Super-VHS videotape. and when they were first permitted to smoke the cigarette. Brief Description of the Three Experiments In Experiment 1. Wertz.60 to about 0. Two of the eight segments were randomly selected to be coded by both raters. n = 7) or heavy (smoke 22-40 cigarettes daily. 1981). Data were collected on eleven women and eight men who were either light (i.

Friesen. which could be accompanied by any of the following: 1+2 (brow raised both medially or laterally). 6 . and 15 frames. 1960. [Agree / (Agree + Disagree]). AU 12 had to receive a minimum intensity rating of “b” using Friesen’s and Ekman’s (1992) updated 5-point “a” to “e” intensity scale if it co-occurred with AU 6. 25 or 26 (jaw parted or jaw lowered) (Ekman. and AU 12 had to receive a minimum intensity rating of “c” if it did not appear with AU 6. 1995. In addition. a minimum intensity rating of “b” was required in order to meet criteria (Friesen & Ekman. 1994. 15 (lip corner depress). 1982. 1986. Two time periods previously found to be especially sensitive to this manipulation (the 10-seconds after being informed of the speech topic and the final 20-seconds before beginning the speech) were coded using FACS. there were 61 segments ranging from 10-20 seconds. 3) AU intensity. & O’Sullivan. 20 (lip stretch). They also were told that this speech would be video recorded and evaluated by psychologists on a variety of psychological variables. and 4) emotion-specified expressions. Ekman.167-186. 1992). 1978) and in published studies referenced below occurred with sufficient frequency to permit analysis. Because some of the participants were informed of the speech topic on two occasions. & Ebert. Coders were said to agree on the occurrence of an AU if they both identified it within the same time window. 1980. Lowery. 1/6th. Facial expressions were recorded using a JVC Super-VHS recorder and Super-VHS videotape. two negative emotion-specified expressions (sadness and disgust) as defined in the Investigator's Guide that accompanies the FACS manual (Ekman & Friesen. pp. These AUs are thought to appear during the expression of negative emotion (Ekman & Friesen. & Ancoli.e. 1995. The reliability of FACS was assessed at four levels of analysis: 1) Occurrence/nonoccurrence of individual AU scoring. Data Analysis In previous research. In addition to individual AUs.Journal of Nonverbal Behavior. press)]. & Friesen. unilateral 14 (dimpler). Smith. 10 (upper lip raise). 1993). To increase stability of estimate. Kappa was used in the present study and is reported below. data were pooled across the three studies prior to computing kappa coefficients. Rozin. we used time windows of 0. 5. In assessing precision of scoring. For negative AUs. For expressions to be considered positive.. Soussignan & Schaal. Fleiss. 1/3rd. 1989). Gosselin. Negative emotional expressions were defined by the absence of AU 12 and the presence of at least one of the following AUs: 9 (nose wrinkle). Friesen. respectively. which correspond to 1/30th. Vrana. 25. coefficient kappa is the preferred statistic (Cohen. and 1/2 second. 1981). Ekman et al. Participants were asked to present a three-minute speech about what they liked and disliked about their body and physical appearance. & Dore. Kirouac. Davidson. 10. because this statistic fails to account for agreements due to chance. 2) temporal precision of individual AU scoring. and 1+ 4 (pulling the medial portion of the eyebrows upwards and together). 1980. As noted above. reliability of FACS AUs was typically assessed as the proportion or percentage of agreement between coders (i. 1988. Sayette & Hufford. Coded as positive were AU 12 (lip corner pull) and AU 6 + 12 (cheek raise with lip corner pull). 1996. Ekman. 1990. we assessed molar positive and negative expressions.

reliability was good to excellent for nearly all (90%) AUs. These included AUs in all regions of the face. 25. As the tolerance window decreased in size. --------------------------------Insert Table 3 about here --------------------------------Discussion Our major finding was that FACS had good to excellent reliability for spontaneously generated facial behavior. 1999).g. Across three experiments. With the exception of AU 5.. the number of AUs with good to excellent reliability decreased.and negative. Intensity scoring of AU 15 was acceptable beginning with a 1/6th second or larger tolerance window and a 3-point but not 5-point intensity scale.167-186. kappas were calculated for the four AUs that were coded for intensity. we were able to include the actions that are most common in studies of emotion and paralinguistic expression (e.AU combinations using the 3-point intensity scale. Inter-Observer Reliability for the Occurrence of Specific AUs and for Precision of Measurement Using a 1/2-second tolerance window. with a ½ second window. Many of these AUs involve 7 . reliability was excellent even at the most stringent (to the frame) level of analysis. negative emotion was examined more specifically. Reliabilities for disgust and sadness are also presented in Table 3.Journal of Nonverbal Behavior. Reliability was better for 3-point intensity scoring than for 5-point scoring (see Table 2). Negative AU combinations also were coded reliably. Kappas remained excellent for disgust and sadness. Results Nineteen AUs met the 48-frame inclusion criterion and were included for analysis.08 with the 5-point scale)]. --------------------------------Insert Table 1 about here --------------------------------Inter-Observer Reliability for Action Unit Intensity To examine the effect of using a 3-point vs. For positive AU combinations..05). [Reliabilities were comparable using 5-point intensity scoring (kappas within . with the exception of the zero frame tolerance window (kappas reduced by . Even at the smallest tolerance window. pp. Cohn et al. 11 of 19 AUs continued to have good to excellent reliability. Intensity scoring of AU 20 was borderline acceptable on the 3point scale.07 to . all but two AUs (AUs 7 and 23) had good to excellent reliability (see Table 1). In addition. --------------------------------Insert Table 2 about here --------------------------------Inter-observer reliability for emotion-specified expressions Table 3 shows the corresponding values for positive. a 5-point intensity scale on reliability. Intensity scoring of AU 10 and AU 12 was acceptable even with a zero frame (1/30th second) tolerance window. however.

is that five-point intensity scoring requires finer differentiation. of course. & Friesen. Reliability also was good for AU intensity. These AUs often co-occur. Even when scoring was to the exact frame. The tightening of the lower lid in AU 7 is a relatively small appearance change that often is mistaken for AU 6. 1995). When brief latencies are crucial to hypotheses. Because AUs 23 and 24 are both associated with emotion-specified anger. however. 25. reliability for these AUs in a large sample of directed facial action tasks. One.Journal of Nonverbal Behavior. Nevertheless.167-186. Moreover. Even a tolerance window of 1/6th second. however. reliability for AU occurrence varies depending on which intensity criteria are followed. AU 23 (lip tightening) is a relatively subtle change in appearance often mistaken for AU 24 (lip pressing). Ekman.versus 5-point scoring was followed. At a minimum. Zlochower. Trace levels of occurrence are counted when scoring 5-point intensity. FACS training and documentation would benefit by increased attention to sources of confusion between those action units for which systematic confusions were found. 1990) smaller tolerance windows may be necessary. was considerably improved when the tolerance window expanded to 1/2 second. Not surprisingly. Only two AUs. A second finding was that reliability estimates varied depending on time frame precision. By evaluating reliability using different tolerance windows ranging from exact frame reliability to reliability within 1/2 second. however (e. while they are ignored when scoring 3-point intensity. had fair reliability even at the slowest sampling rate. reliability remained good. Kappas did not increase much.. 1993. Ekman et al.. Cohn. Similarly. between the 1/6th second and the 1/3rd and 1/2 second windows. these AUs are central to emotion expression and paralinguistic communication. while adequate at the precise frame-byframe unit of measurement. AU 23 and AU 24 are both controlled by the same muscle and co-occur frequently. we found that reliability. 8 . is that what counts as an occurrence differs depending on which intensity scale is used. AU 7 and AU 23. Results also indicated that reliability was excellent for AU combinations associated with disgust and sadness. 1985. confusion between them may have little consequence at this level of description. Lien. 1993. agreement on the 5-point intensity scale was somewhat lower than that for the 3-point scale. A second. however. which is controlled by the same muscle. This can be seen for AU 12. Ruch. Reliability for occurrence of AU 12 was higher when 3.g. pp. and Kanade (1999) found relatively low. provides adequate latitude for temporal agreement. These data are reassuring for investigators interested in assessing emotion-specified positive and negative affect (e. Sayette & Hufford.g. This pattern may reflect difficulty discerning the exact frame that an AU reaches minimum requirements for coding. which makes the distinction difficult as well. though acceptable. Frank. When molar distinctions were drawn between positive and negative emotion-related AUs. There are at least two reasons for this difference. Reliability improved significantly between the 1/30th second and 1/6th second frame tolerance windows. Consequently. a 1/2 second tolerance window is probably acceptable. reliability was good to excellent. the tolerance window used should be noted when reporting reliabilities in studies that include AU duration as a variable. For most purposes. subtle differences in appearance. Hess & Kleck.

Emotion: Theory. Hager. p.). 1999) Saarbruecken..H. Despite difficulties that can arise when coding spontaneous expressions in the laboratory. 168-173. evidence to support reliable use of the most comprehensive facial coding system (FACS) has yet to be published. research and experience. 291-306. Annals of Behavioral Medicine. 9 .edu. Vol 3: Biological foundations of emotion (pp. Nevertheless. Ekman & Rosenberg. Helms.167-186. The reliability of FACS coding was not evaluated across different laboratories. P.L. It is important to establish that different laboratories are using FACS in the same way. 281-296). and a meta-analysis of studies using FACS would be timely. to our knowledge. T.. Bartlett. Another limitation is that the validity of FACS as a measure of emotion was not assessed.S. 253-263. & Mishkin. (1998). T. Siegler. NY: Oxford. provide an important test of the reliability of FACS. Although we included data from three separate studies. Department of Psychology. Reliability studies are needed in which the same data sets are coded by multiple groups of FACS coders. E. 1997). We thank Adena Zlochower and Peter Britton for their helpful comments and David Liu for his technical assistance. 25. FL: Academic Press. (1999). V. There is a large literature relevant to this topic (cf. Acknowledgements This research was supported by National Institute on Drug Abuse Grant R01-DA10605 and National Institute on Alcohol Abuse and Alcoholism Grant R29-AA09918 to Michael Sayette. Correspondence concerning this manuscript should be addressed to Michael Sayette. University of Pittsburgh. M. PA 15260.C. Babyak. References Proceedings of the 8th European Conference on Facial Expression. (1992). Banninger-Huber. & Barefoot. The present data. Germany. In R. Orlando. B.A.. Kellerman (Eds. K.C. Electronic mail can be sent to sayette@pitt. The amygdala: Sensory gateway to emotions. & Sejnowski. Russell & Fernandez-Dols. M. which precluded reliability estimation. I. A. J.& Young. these data empirically confirm that FACS can be reliably used to code spontaneous facial expression. pp.P. 2.J.Journal of Nonverbal Behavior. and Meaning (September. Psychophysiology 36. (1986). (1998). J.C. 20.E. M. Psychotherapy Research. M. In the eye of the beholder: The science of face perception.J. Bruce. 604 Old Engineering Hall. Haney. 1997. Ekman.. Plutchik & H. Measuring facial expressions by computer image analysis.. Aggleton. Brummett.. J. Maynard... Measures of hostility as predictors of facial affect during social interaction: Evidence for construct validity. and National Institute of Mental Health Grant R01-51435 to Jeffrey Cohn. Measurement. some AUs occurred with low frequency. using three different emotion induction procedures. Prototypical affective microsequences in psychotherapeutic interaction. The current study as has some limitations. Pittsburgh.. The coding of facial expression to assess emotional responding is becoming increasingly popular.

Educational and psychological measurement. J. & Friesen. (1978). T.. A. 72. 11. Goldsmith. Mussen Series Ed. & Ancoli.cs.Journal of Nonverbal Behavior. K. A coefficient of agreement for nominal scales. & Russell. & Tomkins. (1986)... Miyake. 269283. The Duchenne smile: Emotional expression and brain physiology II. P. 578-583.. Friesen. Journal of Personality and Social Psychology.W. Human ethology. W. Chesney.. Ekman.. pp. 6. P.E. P. In M. (1990). M. 25. (1972). afterwords. Methods for measuring facial action. Campos. Ekman. (1988). Ekman.H.. 21. Facial affect scoring technique (FAST): A first validity study. 20. J.. Journal of Personality and Social Psychology. Developmental Psychology. W. 3. Friesen. 1208-210. Ekman. (1983). 37-58. (1997).. Psychophysiology.R. and commentaries.M. (1992). Campos.W. (1988). Haith & J. W. Friesen. Journal of Personality and Social Psychology. H. Smiles when lying. Ekman. M. 37-46. P. P. 49. Lamb. S.L. Ekman. 45-90). Type A behavior pattern: Facial behavior and speech components. D. Friesen. 164-176. 783-916). New York: Oxford University Press. Cohn. & Simons. Rationale and reliability for EMFACS Coders.. M. Infant Behavior and Development. Autonomic nervous system activity distinguishes among emotions. Infancy and developmental psychobiology (pp. (1989).... (1992).. Semiotica. Ekman (Eds. Levenson. Facial expressions in Hollywood’s portrayal of emotion.J. 493-505. J. Zlochower. Camras. I. L.V. Automated face analysis by feature point tracking has high concurrent validity with manual FACS coding.. W. & Friesen. J. Camras. Palo Alto: Consulting Psychologist Press. M. Carroll. Socioemotional development. & Bradshaw. 10. P.. (1983). (1982). J. & Hecker. Journal of Personality and Social Psychology. M.167-186. Science. Lien.. A new pan-cultural facial expression of emotion. Expressive development and basic emotions. (1999). Ekman. R.H. R.J. W. Ekman. with introduction. 35-43. W. In K. Eibl-Eibesfeldt. & Kanade. V. . NY: Aldine de Gruyter. & Friesen.) (P. Ekman.. 54. (Ed. (1982). K. Cognition and Emotion. A.cmu.V. 58. W. Scherer & P. P. Is the startle reaction an emotion? Journal of Personality and Social Psychology.V. 39. & Friesen. P. Handbook of methods in nonverbal behavior research (pp.) (1998) Third Edition of Charles Darwin’s The Expression of the Emotions in Man and Animals.V. W. Ekman.J. Psychosomatic Medicine. 414-420.A.. Cohen. Cohn. J. 342-353. & Friesen. & O’Sullivan.V. S. 1416-1426. & Stenberg. 307-319. Cambridge: Cambridge University. Barrett. C. P. London: HarperCollins. Japanese and American infants’ responses to arm restraint. 159-168. & Friesen. 10 . L. P.). Motivation and Emotion.C. G. Oster. Unpublished. 28. P.H. Davidson.. Campos (Eds.). Available from http://www. 36. H. (1985). Ekman. J. P. (1960). W. Black. (1980) Facial signs of emotional experience. J. and Elmore. Effect of contingent changes in mothers' affective expression on the organization of behavior in 3-month-old infants. 1125-1134. 53. M.. Facial action coding system. W.F.S. R.V.. (1990).

Friesen. (1988). (pp.. Differentiating emotion elicited and deliberate emotional facial expressions. Engen. Frank. L. Comprehensive data base for facial expression. Fridlund. (1997). R. Psychophysiology. Journal of Personality and Social Psychology 64. The ability to detect deceit generalizes across different types of high stake lies.J. Gosselin. University of Delaware.). (1985). & Ekman. 11 . P. Izard. Unpublished manuscript. (1979). NY: Wiley. H. C. European Archives of Psychiatry and Neurological Sciences. 89. What the face reveals: Basic and applied studies of spontaneous expressions using the Facial Action Coding System (FACS). (1986). Neurosurgery.167-186.E. Alcohol and stress response dampening: Pharmacological effects. (1969). France.. Ekman.cs. D.G. Katsikitis. M. Heller.. R. (1997). & Davidson. Dougherty. Kanade. E. (1988). Kirouac. 31-34. A study of facial expression in Parkinson's disease using a novel microcomputer-based method. P. 315-326. (1980).E. The behavioral ecology and sociality of human faces. 90-121. Cohn.M.J. & Pilowsky. pp.G. Journal of Abnormal Psychology. pp. (1983).. E. and Psychiatry..The Maximally Discriminative Facial Movement Coding System (MAX). 46-53. Grossman.V. 51. 528-538.H. 369-385. (1982).. pp.. 25. L. W. 362-366. 13. Frank. Proceedings of the 4th IEEE international conference on automatic face and gesture recognition. Behavioral markers and recognizability of the smile of enjoyment. Grenoble. Ekman. (1993). New York. (1992). 22. San Francisco. In Ekman. Ellgring. & Rosenberg. U. and tension reduction. Y.). (1981). Friesen. Patterns of brain electrical activity during facial signs of emotion in ten-month-old infants. F.. Del. G. & Tian. Hager. The asymmetry of facial actions is inconsistent with models of hemispheric specialization. Cited in V. expectancy. J. 236. NY: Academic Press. Ekman. C. & Haynal. Bruce & A. A system for identifying affect expressions by holistic judgments. A. Oxford University Press: New York.W. (1995). Fox.J. Components and recognition of facial expression in the communication of emotion by actors.A. 230-236. Journal of Personality and Social Psychology.G. University of California. (1992). E. M. M. Izard.B. Sher. Journal of Neurology. K. 68. M. N. Newman.W. C. V. Hess. & Ekman. 398-407).. 24. T.M. 307-318.. (2000).Journal of Nonverbal Behavior. Fleiss. Frank. 83-93.. Nonverbal expression of psychological states in psychiatric patients. & Newlin. What the face reveals: Basic and applied studies of spontaneous expression using the Facial Action Coding System (FACS). Available from http://www. M.L. The Judicial Review. NY: Oxford. Hjortsjo. Levenson. Man’s face and mimic language. European Journal of Social Psychology. R.L. 1429-1439. Statistical methods for rates and proportions. (1996). Young (1998). & Kleck. The Perception of Odors. In M. Unpublished Manuscript... T.S. Developmental Psychology. J. 2. J. Journal of Personality and Social Psychology. & Dore.Y. 72. P. P.. & Rosenberg. (1997). Assessing deception: Implications for the courtroom. Clark (Ed. Instructional Resource Center. Depression and suicide faces. Review of Personality and Social Psychology. Changes in FACS scoring.cmu. (1997). P. 83-96. (Eds.).: University of Delaware. J. & Hembree. P. 20. P. (1990).V. In the eye of the beholder: The science of face perception. (Eds. I.F. NY: Oxford.

& Parrott.A. Experimental and Clinical Psychopharmacology. J. National Advisory Mental Health Council. Effects of Olfactory Stimuli on Urge Reduction in Smokers. (1991). D. (1991). Wertz. (1989). Journal of Personality and Social Psychology. (1992). 9. 100. (1995) American Psychologist. (in press). 54 (Serial No.A. 541-545. 50. J. Sayette. Russell... A. & Ross.J. A multi-dimensional analysis of cue-elicited craving in heavy smokers and tobacco chippers.167-186.. D.Z. Rinn. Psychological Bulletin. International Journal of the Addictions. 417-423. & Perrott.S. Shiffman. 1115-1131. Lien. J. NY: Cambridge University. 7. Monographs of the Society for Research in Child Development.E.M. Journal of Studies on Alcohol. B.. Breiner. & Ekman. J. J.F.. T. (1994).. M. C. Is there universal recognition of emotion from facial expression? A review of the cross-cultural studies. J. D. A. S. Feldman & B. Tesman. 151-159. & Wilson. (1992). UK. Experimental and Clinical Psychopharmacology. 102-141.R.R. The effect of alcohol on emotional response to a social stressor. & Wilson.S. C. Journal of Studies on Alcohol.T. M.A. M. Cohn. E. 56-62. M. (1999). What the face reveals: Basic and applied studies of spontaneous expressions using the Facial Action Coding System (FACS).. 28. 33-58.... Abrams. P. W. tracking. Sayette. & Waters. & Monti.. MA: A. E. L. R. Russell.R.A. Oster.A. 483-489. S.. M. Smith. Kanade. M. Rohsenow.. G. Neuropsychology of facial expression. Journal of Robotics and Autonomous Systems.. & Shephard.. Lowery. C. Adult judgments and fine-grained analysis of infant facial expressions: Testing the validity of a priori coding formulas. Intoxication and exposure to stress: The effects of temporal patterning. Matias. (Eds.B. (1995). Hegley.I. W. (pp. P. Urge and affect: A facial coding analysis of smokers. C. The development of emotion expression during the first two years of life. C. Malatesta. (1989).. 31.Journal of Nonverbal Behavior.. Varieties of disgust faces and the structure of disgust. & Li.. (1997). 957-993. Culver. Sayette. 870-881.J. 114. (1997). (manuscript under review). A test of the appraisal-disruption model of alcohol and stress.R. E. & Rosenberg. (1990-1991).. T. Computer facial animation.F. Ruch.. 12 . 25. In R.. (1993). 3.)..A. Developmental Psychology. Fundamentals of nonverbal behavior. 838-845). W.. 66. Detection. H. M. Parke. J.J. Cambridge. J. & Hufford.. M. Peters. L. Rozin. Developmental Psychology.K. D. 25.. M. Personality and Individual Differences. & Fernandez-Dols.. (1994). Childress. R. In Ekman. Cohn. 16. Oxford University Press: New York. G. F. D. (1997). Perrott.. alcohol. The psychology of facial expression. P. 89-102. Extraversion. (2000). K. P. Sayette. 53. 219). M. Rosenberg. Cognition and Emotion. Martin...A. Introduction: The study of spontaneous facial expressions in psychology. Coherence between expressive and experiential systems in emotion.A. and classification of subtle changes in facial expression. Sayette. Niaura.). A comparison of two systems to code infants' affective expression. Rosenberg.S. Wellesley..S. M.C. Sayette. (1994). Martin. 25. Journal of Abnormal Psychology.A. & Nagel.. and enjoyment. 131146. R.J. Wertz. Rime (Eds.M. & Ebert. & Hufford. Cue reactivity in addictive behaviors: Theoretical and treatment implications.M.(1996). pp. J. 318).J.M.

B. R. Developmental Psychology.L. 13 . & Schaal. Dimensions of appraisal in psychological response in emotion. 26.. T. E. Merrill-Palmer Quarterly of Behavior and Development.. Kanade. Children’s responsiveness to odors: Influences of hedonic valence of odor.A. 30. 339-353. Als.167-186. 23.R. (1988). 32. (1996).edu/~face Tronick. Monadic phases: A structural descriptive analysis of infant-mother face-to-face interaction. R. Journal of Personality and Social Psychology.F. & Josephs.M.cs.Journal of Nonverbal Behavior.. Journal of Abnormal Psychology. Psychophysiology. Recognizing action units for facial expression analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence. S. Smith. Y. C. 56. J. Steele. 367-379. (1980). (1993). 25. Vrana. pp. gender. (1989). (in press). 3-24. age. and social presence. C. Soussignan. Tian. H.A. Drinking your troubles away II: An attention-allocation model of alcohol's effect on psychological stress. 97. 279-286. The psychophysiology of disgust: Differentiating negative emotional contexts with facial EMG. B.cmu. & Cohn. Available from http://www. & Brazelton. T. 196205.

64 0.44 0.67 0. nostril dilated by the medial slip of the muscle Description of muscle movement Neutral.67 0.k. Cheeks compressed against teeth 168 0.58 1/6th 0. pars medialis Frontalis. pars orbitalis Orbicularis oculi. Caninus) <48 800 <48 -0.76 0. baseline expression Inner corner of eyebrow raised Outer corner of eyebrow raised Eyebrows drawn medially and down 5 6 7 9 Not Applicable Frontalis.68 0.59 0.74 0. superior part of the nasolabial furrow deepened.82 0. 25.81 11 Levator anguli oris (a.72 0.76 1/3rd 0.70 0. eyes narrowed Lower eyelid raised and drawn medially Upper lip raised and inverted. Depressor supercilii Levator palpebrae superioris Orbicularis oculi.85 0. pp. pars lateralis Corrugator supercilii.66 0.74 -- -0.67 -- -0.71 -- -0.53 0.79 0.167-186.71 0. Tolerance Window (seconds) Frames 5124 386 323 480 201 136 48 0.49 0.83 0.79 0.75 14 .82 0.76 0.a. Table 1.Journal of Nonverbal Behavior.67 1/30 th 0.83 AU Facial muscle 0 1 2 4 Eyes widened Cheeks raised.81 1/2 0. only muscle in the deep layer of muscles that opens the lips Lip corners tightened. pars palpebralis Levator labii superioris alaeque nasi 10 Levator labii superioris 96 0.56 0. Kappa coefficients for single action units.72 0.79 0.76 0. nasolabial furrow deepened producing square-like furrows around nostrils Lower to medial part of the nasolabial furrow deepened Lip corners pulled up and laterally Angle of the mouth elevated.81 0.78 0.69 0.73 0.76 -- 12 13 Zygomaticus major Zygomaticus minor 14 Buccinator Upper lip raised.

61 ---0.65 -0.74 0. relaxed temporal and internal pterygoid Pterygoids and digastric Orbicularis oris Relaxation of levator palpebrae superioris Orbicularis oculi Relaxation of levator palpebrae superioris.68 0.98 0.71 -0.50 0.55 0.a.94 0.38 -0. relaxation of mentalis.60 -0. pars palpebralis Relaxation of levator palpebrae i i bi l i li Wink <48 ---15 -- .76 -0.70 ---0. Triangularis) Depressor labii inferioris Mentalis Incisivii labii superioris andIncisivii labii inferioris Not Applicable Risorius w/ platysma Orbicularis oris Orbicularis oris Orbicularis oris Depressor labii inferioris. pars palpebralis Relaxation of levator palpebrae superioris. 25.64 0.62 0. orbicularis oculi.62 0.Journal of Nonverbal Behavior.47 0.65 0.47 -0.99 0.69 0.63 0.167-186. orbicularis oculi.65 0. pp.32 0.66 0. pars palpebralis Orbicularis oculi.72 -0.53 0.72 -0.67 0.54 0.58 0.41 0.79 ---- 15 16 17 18 Tongue show Lip corners pulled laterally Lips everted (funneled) Lips tightened Lips pressed together Lips parted Jaw dropped Mouth stretched open Lips sucked Upper eyelid droop Eyelid slit Eyes closed Corner of the mouth pulled downward and inward Lower lip pulled down and laterally Skin of chin elevated Lips pursed 19 20 22 23 24 25 26 27 28 41 42 43 44 45 Eyes squinted Blink <48 <48 --- --- --- --- 46 Depressor anguli oris (a.81 0.57 0.65 -0.75 0.79 -0.76 ----0.k.54 -0. or orbicularis oris Masseter. 71 <48 136 57 50 162 <48 63 259 789 696 <48 90 <48 <48 <48 0.

167-186.61 0. pp.69 0.74 0.46 0.57 0.67 0.53 0.79 0.63 0.48 0.49 AU 10 96 12 800 15 71 20 162 16 .59 0. Frames 0.77 0. 25.Journal of Nonverbal Behavior. pars palpebralis Table 2.70 0.72 0. Kappa coefficients for 3.70 0.31 0.68 0.66 0.39 0.45 1/30th Tolerance Window (seconds) 5-Point Intensity Scale 3-Point Intensity Scale 1/6th 1/3rd 1/2 1/30 th 1/6th 1/3rd 1/2 0.61 0.36 0. orbicularis oculi.57 0.44 0.59 0.60 0.77 0.63 0.65 0. superioris.53 0.and 5-point intensity scoring.74 0.

Results shown for 3-point intensity scoring.78 Negative emotion 313 .86 .81 .47 .75 .61 Note. Kappa coefficients for emotion-specified combinations Tolerance Window (seconds) 1/30th 1/6th 1/3rd Frames 1/2 Action Unit Aggregates Positive emotion 335 . 25.82 Sadness 37 .67 .82 .73 17 .Journal of Nonverbal Behavior.79 . Table 3.85 . .83 .64 .74 Disgust 103 .71 .167-186. pp.