Tactile Hand Gesture Recognition through Haptic Feedback for Affective Online Communication

Hae Youn Joung and Ellen Yi-Luen Do
College of Architecture Georgia Institute of Technology 247 4th St Atlanta, GA 30332, USA {joannejoung, ellendo}@gatech.edu

Abstract. Our study explores how individuals communicate emotions using tactile hand gestures and provides evidence supporting the link between emotions and gestures to investigate the usability of tactile hand gestures for emotional online communication. Tactile hand gestures are used as the source of information to get to emotions. In this study, behavioral aspects of tactile hand gestures being used for emotional interaction are observed through a sensory input device and analyzed using analysis of variance (ANOVA). In user experiments, subjects perform tactile hand gestures on the sensory input device in the response of a list of distinct emotions (i.e. excited, happy, relaxed, sleepy, tired, lonely, angry and alarmed). An analytical method is used to recognize gestures in terms of signal parameters such as intensity, temporal frequency, spatial frequency and pattern correlation. We found that different emotions are statistically associated with different tactile hand gestures. This research introduces a new way of creating online emotional communication devices that approximate the use of natural tactile hand gestures in face-to-face communication. Keywords: Tactile hand gesture recognition, affective communication, haptic interface, and tactile stimulation.



Touch, central to emotional communication, is the simplest and the most straightforward of all sensory systems [1] and it has been described as the most fundamental means for people in contact with the world [2]. It is also capable of communicating and eliciting emotions [3]. Tactile gestures exhibit the natural capability and tendency of humans to move their hands to express and communicate emotions. In particular, in emotional communication, tactile hand gestures arouse human emotions through touch. In face-to-face communication, gestures cover a wide range of non-verbal communication including body language, facial expressions, hand gestures, and sign language. Among these various types of gestures, tactile hand gestures are one form of communication using the sensory modality for touch. Tactile hand gestures are a natural social tool of expressive behavior that describes emotions and situations in human social life. A gesture is a motion of the body that

HCI International 2011, Volume 6, LNCS 6766, 2011. © Springer-Verlag Berlin Heidelberg

from this research we expect that the relationships between touch hand gestures and emotions are determined. we suggest applying this research to prototype called emotion communication device in future work section. researchers have also employed various sensory modalities that enable affective online communication through interactive media. We expect prototype validations show that the gesture-to-emotion conversion can be used to enable the emotional interaction between humans in distance through a new media. voice. Such tactile hand gestures are commonly used in face-to-face communication. We can communicate the emotion through the ‘soothing act’. In these studies. Other designs have snooze buttons with pressure sensors that elicit pushing. Yet these expressive actions offer no feedback to a person [8]. The aim of this study is to better understand the relationship between hand gestures and emotions. These emotional communication techniques. To enable on-line communication that approximates a face-to-face interaction. researchers have developed a wearable tactile interface that attempts to encourage humans to share their emotions seamlessly through online digital communication methods. © Springer-Verlag Berlin Heidelberg . however. Their use. is limited in the era of online communication environment. Yet the design needs additional feedback to communicate understanding.contains information [4]. In contrast. Literatures focus tactile interface and haptic device. Our research focuses on the study of tactile hand gestures for the use in affective online communication. tactile hand gestures are a natural and intuitive way to express personal emotions and situations [9]. Especially. 2011. [6]. human computer interaction (HCI) and affective computing researchers have converted emotional messages into the form of text. 'pin alarm' by Hellman and Ypma allows for setting the waking up time with meaningful expressive actions. Other researchers have designed and developed a mobile emotional messaging system named eMoto to enable affective loop experiences [7]. LNCS 6766. Recently. We are interested in determining how emotions affect the motion of tactile hand gestures and how we can code different emotions with tactile hand gestures in devices for online communication. Method of expressing emotions through hand gesture: expressive gestures are used as a method of expressing emotions through hand gesture to control a device. require an explicit process of conversion from emotions to predefined formats such as human/computer languages and symbols. stroking or slamming. soothing someone’s arm is a tactile gesture. and video [5]. the authors have discussed an integrated physical. Volume 6. The previous studies of affective computing indicate that the aim of communication should be to design embodied interaction that harmonizes with our everyday practices and everyday bodily experiences. intellectual and social experience to incorporate the aspect in design for emotional interaction. The authors argue that emotions and actions are closely intertwined and provide an answer to the following question: can we communicate and decode distinct emotions with tactile hand gestures through on-line communication devices in a similar way that we do in face-to-face communication? HCI International 2011. however. Finally. For example. In addition. By pushing as many pins as possible you indicate that you want a lot of sleep and by pushing them one by one you indicate a more urgent situation.

The list of distinct emotions is defined by Russell’s dimensional model of emotions to be described in Section 2. To analyze this relationship. sleepy. happy. spatial frequency and pattern correlation.1 Type of Emotions Theory of emotion is mostly based on cognitive psychology but with contributions from learning theory. Proposed wearable tactile interface device facilitating touch sensor arrays. relaxed. shaking and squeezing.2.1 [9]. 1. especially those performed with the fingers. and other disciplines including HCI International 2011. 2011. © Springer-Verlag Berlin Heidelberg . angry and alarmed) requested by the researchers. Distinct tactile hand gestures are determined by a combination of these four signal properties to be described in Section 2. lonely. physiological psychology. we obtain tactile hand gesture signals by asking participants to express their emotions while they are holding and interacting with a wearable tactile user interface device shown in Fig. tired. temporal frequency. LNCS 6766.e. Russell’s model is widely used as the means of emotion classification in the fields of emotional research and affective science. Volume 6. excited. Fig. The study results showing the relationship between the two are summarized in terms of a look-up table to be used in potential applications such as online emotional communication. 2. The device imitates the shape of a human arm so that subjects in the experiment perform their hand gestures as they express their emotions on human arm such as dragging. The participants are asked to perform tactile hand gestures on the interface device in response to distinct emotions (i. The tactile interface device records hand gestures exerted on the device in terms of signal parameters such as intensity. We analyze the recoded hand gestures and their corresponding emotions using a statistical analysis method called multivariate analysis of variance (MANOVA) to find the statistical relationship between the participants’ emotions and their tactile hand gestures. 1.2 Approach The aim of this study is to analyze the relationship between human emotions and corresponding tactile hand gestures.

Pattern correlation: the vector correlation between input gestures and pattern symbols such as circle. Russell’s dimensional model of emotions. 2011. To define the types of tactile hand gesture. Arousal: mental state of being awake or reactive to stimuli. Fig. named the circumflex model of affect. etc. line. hand gestures. 2. • • • • Intensity: the degree of contact pressure. the emotional states of the subjects in our research are defined based on Russell’s dimensional model. which include body language. we propose a new model that consists of four tactile signal properties as summarized below. LNCS 6766. Spatial frequency: the number of occurrences of contact per unit distance. and sign language. Volume 6. symbolic or representational cues revealing cognitive properties [12]. facial expressions. HCI International 2011. Among many proposed models of emotion. In this model.1 Model of Tactile Hand Gestures Gestures are defined as nonverbal phrases of actions [11].philosophy [10]. Russell provides the mental map of how emotions are distributed in a two-dimensional system of coordinates where the y-axis is the degree of arousal and the x-axis is the valence and categorizes emotions in terms of pleasure and arousal. © Springer-Verlag Berlin Heidelberg . Gestures involve a wide range of nonverbal human communication. as shown in Fig 1. cross. describing explicit. • • Pleasure: mental state of being positive. Temporal frequency: the number of occurrences of contact per unit time. 2.

The controller board is used to transfer the x-y axis and intensity data of contact points to analysis software. which are shown in Fig. 3. sleepy. and pattern correlation.3 Mapping between Emotion and Tactile Hand Gesture This study is to observe which type of hand gestures the participants perform in a particular emotional situation and to obtain statistically significant data that prove a possibility of tactile hand gestures’ communication in online environment. Participants’ tactile hand gesture inputs in experiments are critical for the analysis of human emotions and furthermore for developing our tactile communication device.2. The resultant data are applied to finding the model-based statistical parametric mapping between the emotional domain and the hand touch gesture domain. temporal frequency. 3 Experiments Behavioral aspects of tactile hand gestures being used for emotional interaction are observed and analyzed using a wearable tactile interface device shown in Fig. Volume 6.e. The device facilitates a multi-touch panel (touch area dimension: 52 x 35 mm ) and its associated electronics (controller board). excited. In Section 2. Multi-Touch demonstration and evaluation kit. Based on these models. 3. 1. The groups in the gesture domain are mapped to those in the emotional domain. The wearable tactile interface device is designed to recognize gestures in terms of various sensory parameters such as intensity. As we aim to make the subjects emotionally involved in a physical sense. HCI International 2011. the model parameters of emotions and gestures are provided. 2 Fig. it is important that the gestures we pick are not singular. tired. © Springer-Verlag Berlin Heidelberg . The obtained pressure data is analyzed for the correlation between emotions and gestures using the analysis software. relaxed. LNCS 6766. In the response of a list of distinct emotions (i. lonely and angry). iconic or symbolic gestures. subjects perform tactile hand gestures are performed on the tactile interface device. 2011. happy.1 and Section 2. spatial frequency. analysis of variance (ANOVA).2. but gestures that give rise to a physical experience that harmonizes with what the user is trying to express. the statistical relationship between emotional parameters and corresponding gestures is found by using the collection of statistical models.

Based on the pressed area. The researcher introduces to participants the Russell’s dimensional model of emotions. (5) tired. Multi-finger movements are tracked with cursors. hitting. Participants are selected from those who actively use hand gestures to express their emotions. (4) sleepy. 3. 2011. 4 are expressed by applying finger gestures to the tactile interface device. 35 are female and 35 are male out of 70 participants. Volume 6. 10 seconds are given for each gestural expression and participants are asked to express the eight types of emotion. Fig. (2) happy. LNCS 6766. pummel. (3) relaxed. Eight distinct emotions indicated in the Russell’s emotion chart. The wearable tactile sensor interface with pressure sensor arrays implemented on its surface is used as a sensory input device. who range in age from 20 to 40 years old (23. 5 shows finger pressure points detected by the tactile interface device that measures pressure by detecting contact area. © Springer-Verlag Berlin Heidelberg . Eight distinct states of emotion indicated on the Russell’s emotion chart shown in Fig.Fig. from libraries at Georgia Institute of Technology and public libraries in Georgia. and squeezing). (6) lonely. Each emotion: (1) excited. The experiment takes 10 minutes per participant. The tactile interface device is also available to monitor the position of finger contact points. HCI International 2011.1 Setup The research sample set consists of 70 participants. leads participants to express the emotions which are analyzed by parameters in Russell’s dimensional model (Participants may perform tactile hand gestures such as soothing. the sensor calculates the pressure data in 256 levels. (7) angry and (8) alarmed. 4. The obtained pressure data from participants’ gestures are then analyzed using ANOVA.15 years old in average).

p = 4. p = 1. are related with gestures with high intensity and high temporal frequency. In Fig. As shown in Fig. (a) (b) (c) (d) HCI International 2011. 7(b). To assess the relationship. the tactile gesture corresponding to angry emotion contains the mean value of intensity significantly higher than that of the other emotions. the distributions of gesture samples in spatial frequency do not vary much among various emotions. Each 2-dimensional scatter plot shown in Fig. 7(a). sleep.Fig. 7(c) shows that the emotion group of relaxed.88e-13. © Springer-Verlag Berlin Heidelberg . Volume 6. 7 displays the gesture sample data in two tactile signal properties out of eight. Very small p values for all the four signal properties that are obtained from ANOVA (p = 2. 3. tired and lonely is associated with patterned gestures such as drawing circles and lines. p = 3.76e-111. we determined the correlation between the signal properties of tactile gestures and the emotional states by performing ANOVA for recorded tactile data. which is the future work of this study. The balanced one-way ANOVA of the obtained tactile gesture sample data was performed for the signal properties: (a) intensity. gesture samples associated with happy emotion are placed low in spatial frequency as compared to those with excited. and (d) pattern correlation.08e-82. (7) angry and (8) alarmed. Contact area based pressure detection of the touch sensor panel. However. The mean value of temporal frequency corresponding to alarmed emotion is higher than that of the other emotions. Fig. (c) spatial frequency.99e-4) indicate that at least one emotional sample mean is significantly different from the other emotional sample means. which corresponds to high arousal according to Russell’s dimensional model. 5. this emotion group can be distinguished from the others. By observing pattern correlation values. (2) happy. However. this procedure requires detailed pattern analysis for accurate decoding. The type of emotion whose sample mean is significantly different from the other can be statistically distinguished and decoded from the other emotional states by observing the corresponding property of tactile gesture sample data. As shown in Table 1. the emotion group of (1) excited.2 Results The user research using the wearable tactile sensor device provides us evidence of the relationship between tactile hand gestures and emotions. 2011. However. LNCS 6766. (b) temporal frequency. angry and alarmed. spatial frequency and pattern correlation show less significant distinction in their mean values among the various emotions.

which contains high arousal and pleasure. © Springer-Verlag Berlin Heidelberg . which corresponds to high arousal and displeasure according to Russell’s dimensional model. In addition. the duration of hand HCI International 2011. (2) happy. LNCS 6766. (5) tired. (4) sleepy. (3) relaxed. corresponds to tactile signals of high spatial frequency and high temporal frequency. spatial frequency. (c) Tactile gesture scatter plot: intensity vs.(a) Contact area based pressure detection of the touch sensor panel. Fig. The emotional state of anger. (6) lonely. (b) Tactile gesture scatter plot: temporal frequency vs. 6. and (8) alarmed. is related to tactile signals of high strength and low temporal frequency. The results also show that the emotional state of happiness. Volume 6. Tactile gesture scatter plot: (1) excited. pattern correlation. 2011. (7) angry.

Geldard. S. Laurel.1 2.1 " 0. 113--138.21 0. Mountford (Eds.19 " 0. From user experiments.: Gestures in Human-Computer Communication. the types of tactile hand gestures are categorized in terms of tactile signal properties and related to distinct emotions that the gestures originate from.7 1.).024 0.21 0. 309--317. In: B.040 0.15 Mean and standard deviation of tactile gesture samples.18 0. Determinant of infant behavior II.23 0. pp. LNCS 6766. MA: Addison-Wesley Publishing. (1990) 5.39 " 0. The findings from this investigation show opportunities and promises to use tactile hand gestures for the communication of emotions among people using online digital communication devices at a distance. However.26 0.12 0.030 0.17 0.M. London: Methuen & Co (1963) 4.13 0.4 1.24 0. the Foundations of Embodied Interaction. 1583-1588 (1960) 2. P. we found that different emotions are statistically associated with different tactile hand gestures.: Where the Action is. 102--110 (1972) 3.18 0. ! 1.H. Science. P.67 Spatial Freq.54 0.81 0.044 0. pp. the aspect of pattern correlation as one of tactile gesture properties needs to be further investigated to uncover more gesture information and to better define gesture-emotion relation.19 0. Kurtenbach.12 3 Conclusion In this research.). © Springer-Verlag Berlin Heidelberg .50 0.049 0.3 1.58 0. In: B. pp. ! 2. pp.11 0.72 0. G.4 1.19 0.31 0.074 0. Nursing Research.29 0.21 0.052 0. Cambridge.16 0.31 0. F.J. Table 1. K. Barnett.A.7 2.40 0.8 1. Dourish.054 0.17 0. Hulteen. The Art of Human-Computer Interface Design. References 1. 21.. the relationship between touch hand gestures and emotions will be applied to emotional online communication devices.55 0.53 0.55 0.37 1. 131.19 0. vol. Volume 6.90 0. vol.41 0. A.34 0. MIT Press. Intensity ! Excited Happy Relaxed Sleepy Tired Lonely Angry Alarmed 0.20 0.44 0.: Some neglected possibilities of communication.69 0.5 2. and a prototype design of wearable online tactile communication devices will be demonstrated. 2011. E.3 0.2 " 0. In future study. Wolff. Foss (Ed.: Observations on the early development of smiling.gestures is extended when people express amplified emotional states compared to subtle emotional states. Temporal Freq.3 3.19 0..62 0. MA (2001) HCI International 2011.69 Pattern Correlation ! 0.: A theoretical construct of the concepts of touch as they relate to nursing.

28.: Bridging emotion theory and neurobiology through dynamic systems modeling. Golden-Meadow. and techniques (2000) 9. T. K. (1988) 12. 2011. Behavioral and brain sciences. Cambridge.: Body Language. J.: Aesthetic movement interaction: designing for the pleasure of motion. S. Human-Computer Studies (2007) 8. © Springer-Verlag Berlin Heidelberg . Sweden (2006) 7. S.. LNCS 6766. Fast. MA. Lewis. Journal of personality and social psychology. Int. Djajadiningrat. Belknap Press of Harvard University Press (1999) HCI International 2011. New York: M. Numerical Analysis and Computer Science. Evans & Company. Volume 6. methods. Doctoral Thesis. J.A. Hit Me and I Know How You Feel: A Design Approach to Emotionally Rich Interaction.: Touch Me.: A circumflex model of affect. J. KTH. S. J. 34 (1980) 10. 3rd conference on Designing interactive systems: processes.: Hearing Gesture: How Our Hands Help Us Think. vol.6. M. Petra. 169--245 (2005) 11. Russell.: In situ informants exploring an emotional mobile messaging system in their everyday practice. vol.. Overbeeke. Stockholm. Wensveen. practices. Moen.