HUMAN EMOTION CLASSIFICATION BASED ON DISCRETE WAVELET TRANSFORM AND FUZZY C CLUSTERING

Siddhartha Singh Chandel Sushil Chandra Shailesh Gupta 1 D D U Gorakhpur university, Gorakhpur 2 Institute of nuclear medicine and allied sciences, Delhi 3 C C T Rajasthan University, Rajasthan
1 2 3

ABSTRACT Emotion is one of the most important features of human, without the ability of emotions processing, computers cannot communicate with human in natural way. The main objective of this project holds three important aspects to review the previous works on human emotion detection using Electroencephalogram to design a visual induction based data acquisition protocol for data collection and to propose the new time-frequency analysis based features for emotion detection. In order to perform this assessment, we have collected the 14channel EEG signals (Emotiv) from IAPS (INTERNATIONAL AFFECTIVE PICTURE SYSTEM). There are two to twenty basic emotions defined by different authors. The basic emotions we used are Happy, Disgust ,Calm ,Surprise ,Fear We process this project in the following ways: (a) origin of emotions (b) previous studies on emotion assessment (c) EEG measurement methods (d) pre-processing (e) feature extraction methods (f) artificial classifiers and (g) applications and limitations. We used Discrete Wavelet Transform (dwt) to extract the most salient features from the EEG signal. For classifying the emotion detection we used REE (recurssing energy efficiency) of different channel in FUZZY C-MEAN, REE feature has efficiently clustered the emotions so we get better results of classifying emotions as compare to energy used in FUZZY C-MEAN.

CHAPTER 1 Introduction

1.1 Emotion
Emotion is an important aspect in the interaction and communication between people. Even though emotions are intuitively known to everybody, it is hard to define emotion.

Figure 1.1: Different emotions The Greek philosopher Aristotle thought of emotion as stimulus that evaluates experiences based on the potential for gain or pleasure .Years later in the seventheenth century , Descartes considered emotion to mediate between stimulus and response .Now days there is still little consensus about the definition of emotion . Kleinginna and Kleinginna gathered and analysed 92 definitions of emotion from different definitions and suggested the following comprehensive definition. Emotion is a complex set of interactions among subjective and objective factors, mediated by neural/hormonal system, which can 1. Give rise to affective experiences such as feelings of arousal, pleasure/displeasure; 2. Generate cognitive processes such as emotionally relevant perceptual effects, appraisals, labelling processes; 3. Activate widespread physiological adjustments to the arousing conditions; and

4. Lead to behaviour that is often, but not always, expressive, goal directed, and adaptive. This definition shows the different sides of emotion. On the one hand emotion generates specific feelings and influences someones behaviour. This part of emotion is well known and is in many cases visible to a person himself or to the outside world. On the other hand emotion also adjusts the state of the human brain, and directly or indirectly influences several processes. In spite of the difficulty of precisely defining it, emotion is omnipresent and an important factor in human life. People's moods heavily influence their way of communicating, but also their acting and productivity. Imagine two car drivers, one being happy and the other being very mad. They will be driving totally different. Emotion also plays a crucial role in all-day communication. One can say a word like `OK' in a happy way, but also with disappointment or sarcasm. In most communication this meaning is interpreted from the tone of the voice or from non-verbal communication. Other emotions are in general only expressed by body language, like boredom. A large part of communication is done nowadays by computer or other electronic devices. But this interaction is a lot different from the way human beings interact. Most of the communication between human beings involves non-verbal signs, and the social aspect of this communication is important. Humans also tend to include this social aspect when communicating with computers. The interaction between humans and computers is done in several ways. Most people use simple keyboard and mouse to give instructions to the computer, and that works fine for most tasks. Newer forms of human computer interaction have been invented. The quality of speech recognition by computers is improving every day and becoming more and more common in consumer products. Arm movement is used when playing games on Nintendo's new game console, the Wii.

Figure 1.2: The human brain In addition to these existing ways of interacting with computers, the use of brain activity is becoming increasingly popular. This method is called a brain computer interface (BCI). In theory, everything somebody does or says has its origin in his brain, and when using that activity for human computer interaction, many other information could be used, and the possibilities seem endless. Brain computer interfaces have been constructed to be used as a spelling device , for playing video games or even as a replacement for controlling a windows

computer (combined with electromyography (EMG)) , but the list of possible applications that could be controlled by a BCI includes actually all programs known today.

1.2 Cognitive theory of emotion

Emotion and cognition are two major aspects of human mental life that are widely regarded as distinct but interacting. Emotioncognition interactions are intuitively intriguing and theoretically important. However, there are many different kinds or forms of interaction, and in principle these differ widely in what they imply about the functional organization of the brain. Some forms of interaction are diffuse or nonspecific, whereas others are more nuanced and imply a greater complexity in brain organization. The typical folk psychological view of emotioncognition interaction is nonspecific: for better or worse, emotional states diffusely impact higher cognition (e.g., pleasant emotions are non-specifically beneficial, whereas unpleasant emotions are nonspecifically detrimental). A nonspecific view is parsimonious: complexity should not be posited without compelling evidence. Here, we report neural evidence for a strong highly constrained form of emotioncognition interaction. These data reveal a complexity that implies an integration of emotion and cognition in brain organization.

1.3 Emotion recognition using brain activity

Using a BCI to measure emotion could improve the quality and effectiveness of human computer interaction. This communication can be done in a much more natural way and open up new ways for computers to understand human behaviour or meaning. A lot of research is already done after recognizing emotions. For example, research has been done to make computers recognize emotion from speech, facial expressions or a fusion of both methods. Another method of measuring human emotion is using physiological signals. These physiological signals are available at any moment, and it has been shown that emotional markers are present in electroencephalograms (EEG), and can be deduced from heartbeat rates and skin conductivity. Using these physiological signals also has the advantage that they can hardly be deceived by voluntary control and are available all the time, without needing any further action of the user. The disadvantage of using these signals is that the user has to wear some measurement equipment. This equipment could be very simple when measuring only heartbeat, but EEG measuring devices tend to be more demanding. This method using EEG signals can however still use a lot of improvement. Earlier research has shown that these types of human computer interfaces do not perform very well. They are able to recognize emotion correctly from around 60-70% of the samples given when recognizing three emotions. Some researchers suggests there is a lot of improvement possible when looking at different features or using other methods or techniques. A fusion of different methods could be another solution

1.4 Societal relevance and possible applications

Researching emotion recognition is done by many people. As explained before, many efforts is put into emotion recognition from other modalities, such as speech, text or facial expressions. Emotion recognition from EEG signals is expected to outperform these modalities, since brain activity has direct information about emotion, where other modalities

are an indirect reaction of the emotion. Moreover, EEG signals can be measured at any moment, and are not dependent on someone speaking or generating a facial expression. Possible applications of this technique are many. First of all, the simple task of recognizing someones emotion automatically could assist therapists and psychologists in doing their job. Other applications can be found in the field of human machine interaction, human communication through computers and in assisting disabled people with communicating. In the field of human machine interaction, the computer could adjust its behaviour depending on the users emotion. When a game becomes too boring or too exciting, the game level can be changed. Another example might be that the computer changes the music or the windows background depending on the mood of the user sitting behind it. These types of human computer interaction are already used in the commercial world Communication between people through instant messaging is very common nowadays. This communication is unfortunately only capable of communicating through words, although speech and video are a great addition. The enormous use of smileys in many conversations shows the desire to share and express emotion while talking to others. A brain computer interface could be a useful tool to automatically add emotional information during the conversation. Another application is emotion expression for people suffering from severe muscle diseases like amyotrophic lateral sclerosis (ALS). These patients may not be able to move their muscles or to speak. Some progress is made in creating a brain-computer interface for those patients to enable them to communicate again. A way to express emotion would be a great improvement to such a system. See for more information.

1.5 Research questions

The objective of this project is to create a system for emotion recognition using brain activity and assess the quality of this emotion recognition in practice. This goal can be broken down into several research questions: 1. Is emotion recognition from EEG signals possible in practice? Literature suggests that emotion recognition should be possible from EEG signal. However, the results in practice show very different results. We will design, build and test a system for the offline recognition of emotions from EEG signals based on literature from earlier research. After that we will create a dataset with EEG signals, which can be used to test the quality of emotion recognition. Using this dataset we will also investigate the quality of different parameters for our program. (a) How to perform EEG measurements? We will have to find out how to record EEG data. Using that knowledge, we will setup an experiment to make EEG recordings with known emotion. These recordings will help in training and testing our program. (b) How to process EEG data? Measuring EEG data results in huge amounts of data, containing brain activity, but also a lot of noise and artifacts. We should find out how to handle the EEG data in our program in such a way, that we remove the noise and artifacts and keep the brain activity.

(c) How to recognize emotion from the EEG data? Besides the removal of unnecessary part of the signal, we should decrease the size of the dataset by extracting only the useful information. From this information, the program should be able to detect the emotion of the user at that moment. We should find out how to do that. 1.6 Scope and limitation The program we will create is specifically meant for research purposes, to see whether emotion recognition from brain activity is feasible at this moment. Researchers can use this program to test several methods or techniques easily. For that reason, there will be focus on the techniques and the extendibility, not so much on the user interface. However, while designing the software we keep in mind that the program should be easily extendable to include other methods or algorithms. While this program works on offline data, the ultimate goal is to build an online version of an emotion recognition system or even a hybrid system where vocal, facial and physiological signals are combined. 1.7 Methods In order to accomplish the goal mentioned in the previous section, we have to take several steps: 1. Literature research The first step will be to gather knowledge about emotion, physiological measures such as EEG and brain computer interfaces (BCI). This information includes theoretical information about emotion and the way emotion works in the human brain, as well as information about the measurement of brain activity. Other literature will be consulted about signal processing and data mining, in order to use the latest techniques. 2. Designing and implementing system to analyse emotional EEG signals The next step of the process involves designing the software system that will be able to recognize emotion. The design will be based on earlier attempts to build an emotion recognition system. In this step, the structure of and the techniques we use in our system will be chosen and explained, as well as the way to represent emotion. Based on the design of the system as chosen in the previous step, the system will be implemented for research use. The system will include a very simple user interface. After implementation, the different steps of the system will be tested. 3. Data acquisition We will have to teach the system how to recognize emotion from the brain activity. For this purpose, we need some measurements of brain activity, as well as the emotion someone has at that moment. 4. Training and testing The gathered data will be used to teach our system the details about recognizing emotion from the EEG signals. We will use a part of the data for this training, and another part of the data for testing whether the learned relationship between the EEG signals and emotion is

correct. We will change various parameters and choices we made, in order to find out the best working program. 1.8 OUTLINE The all work is supported by Bio-medical engineering Department, Institute of nuclear medicine and allied sciences, DRDO (Ministry of Defence, India). This work is a part of major project VE-CARE (Virtual Environment- Cognitive Assessment, Rehabilitation, and Enhancement). Chapter 2 will provide information on the theory behind emotion and ways to measure brain activity. In Chapter 3 we will give the information about Emotiv and how to use this, introduction to MATLAB and about IAPS images. Chapter 4 will explain the design of our system, with what features are extracted and classifier used in our project. Finally Chapter 5 gives the information of result and conclusion.

CHAPTER 2 LITERATURE

For recognizing emotion from brain activity, we need information about how the brain handles emotion, and information on how to measure the brain activity. Several methods on measuring brain activity are described in section which will give information about literature on emotion, how emotion is handled by the human brain and emotional models. Section 2.3 will also provide an overview on earlier literature on emotion recognition using brain activity.

2.1 Measuring brain activity The human brain is the part of the body that regulates almost all of the human activity. Therefore the activity of the brain contains a lot of information about someone. Several ways of measuring this activity exist, such as positron emission tomography (PET), functional magnetic resonance imaging (FMRI) and electroencephalography (EEG). 2.1.1 Positron emission tomography With positron emission tomography (PET), a radioactive isotope is injected into someones blood. Because this isotope emits positrons, and is taken with the blood flow, a machine can measure the blood flow. Since it is believed that the blood flow in the brain is highly correlated with brain activity, the machine can show brain activity.

Figure 2.1: Example of a PET scanning system. (Image from www.radiologyinfo.org)

PET is able to measure activity with a high spatial resolution, but it has a low time resolution and a time delay due to the time it takes before the radioactive material has arrived in the brain. Another disadvantage of this method is the radiation a person is subjected to. 2.1.2 Functional magnetic resonance imaging Functional magnetic resonance imaging (fMRI) is another method that depends on the blood flow. Active neurons are known to consume oxygen that is carried by haemoglobin.

Figure 2.2: Example of an fMRI scanning system. (Image from www.ftcm.org.uk)

This consumption of oxygen changes the magnetic properties of haemoglobin, and these magnetic properties are measured by the fMRI system. So, indirectly the brain activity is measured. Just as pet, fMRI has a high spatial resolution, but a low temporal resolution. Moreover, the equipment needed is very expensive.

2.1.3 Electroencephalography Electroencephalography (EEG) uses the electrical activity of the neurons inside the brain.When the neurons are active, they produce an electrical potential. The combination of this electrical potential of groups of neurons can be measured outside the skull, which is done by EEG. Because there is some tissue and even the skull itself between the neurons and the electrodes, it is not possible to measure the exact location of the activity. For EEG measurements an array of electrodes is placed on the scalp. Many caps available use 19

electrodes, although the number of caps using more electrodes is rising. The electrodes are placed according to the international 10-20 system, as discussed in section 2.1.4 this system makes results from different research easily comparable. The brain activity can be coarsely divided into different classes. This division follows the way the activity is produced by the brain. Rhythmic activity The neurons of the brain produce together a rhythmic signal that is constantly present. This signal can be divided into several bands, based on the frequency. Delta band The delta band is the frequency band up to 4 Hz. Delta activity is mainly seen in deep sleep.

Theta band The theta band consists of frequencies between 4Hz and 8Hz. This activity can be observed with drowsiness or meditation

Alpha band The alpha band is the so-called `basic rhythm' and contains the frequencies between 8Hz and 13Hz. It is seen when people are awake, and is known to be more apparent when eyes are opened.

Beta band The beta band contains frequencies between 13Hz and 30Hz. This band is apparent with active thinking or concentration.

Gamma band The gamma band contains frequencies between 30- 100 Hz. This band is representing binding of different populations of neurons together (to do two different things on the same time).

2.1.4 The standardized 10 20 system The standardized 10 20 system is often used to record spontaneous EEG on the surface of the scalp. The size of the cranium of each person is different. Thus a relative system was defined, the standardized 10 20 system.

Figure 2.3: the international 10 20 system seen from (A) left and (B) above the head. A = Ear lobe, C = central, Pg = nasopharyngeal, P = parietal, F = frontal, Fp = frontal polar, O = occipital. The distance between naison and inion is measured and defined as 100%. From the position of the naison in direction to the position of the inion, 6 steps are done to place electrodes:

Beginning with a step of 10%, followed by 4 steps of 20% and ending up with a step of 10%. This movement is done in transversal and median planes, as you can see in the above figure

Figure 2. 4: (C) Location and nomenclature of the intermediate 10% electrodes, as standardized by the American Electroencephalographic Society (Redrawn from Sharbrough, 1991) Bipolar or unipolar electrodes can be used in EEG measurements. With the bipolar electrodes the potential difference between a pair of electrodes is measured, with the unipolar electrodes the potential of each electrode is compared either to a neutral electrode or to the average of all electrodes.

Figure 2.5: bipolar and unipolar recording technique

2.1.5 Conclusions fMRI and PET have a good spacial resolution, but are very slow, as they depend on changes in composition of blood. EEG on the other hand gives immediate responses, but is not able to reflect the exact location of the brain activity. Because of this high time-resolution, the simple procedure and its low cost, EEG is used in our project to measure brain activity.

2.2 Emotion As explained in the introduction, emotion is a concept that is hard to define. Scientists agree on the fact that the human brain is the main source of emotion. However, the precise role of the brain is not clear. Many theories exist about the exact involvement of the brain. 2.2.1 The emotional brain For years, scientists have reasoned about and researched emotion. One of the first theories proposed was the James-Lange theory, in the end of the 19th century. This theory states that emotion is the experience of some changes in the body. For example: when one sees a bear, he will run away, and the running away will provide him the feeling of fear. In the 1920s Cannon and Bard challenged this theory; among others because bodily changes are too slow to be responsible for emotions and artificial body changes are insufficient to generate emotions. They state that the hypothalamus is the brain region that translates stimuli into emotions. Some years later, Papez augmented this idea and made a nice scheme of how emotion would be processed in the brain. It consists of two streams, the stream of feeling and the stream of thinking. He claimed that emotional experiences are a function of activity of both streams that is computed in the cingulated cortex.

Figure 2.6: The Emotional Brain

Many of this theory, especially the pathways he proposed, are proven to exist, although not all the regions he mentioned are important in emotion processing. The coloured part is considered part of the limbic system in 1949; Maclean introduced another, more accurate model, and invented the term limbic system. He claimed (as a neo-Jamesian) that perception of the world, combined with information on body changes generates emotional experience, and this integration of knowledge occurs. Some of these structures are: Amygdala The amygdalae are two groups of neurons deep inside the human brain. Together, they are one of the most important brain regions for emotion. Together they are called the amygdala. One of its functions is to interpret the emotional value of incoming signals. When a person receives some sort of signal with some emotional load, it is recognized by the amygdala. Another function of this organ is fear conditioning: it helps in learning a connection between some stimulus and a threatening event. For example, when a rat hears a sound and gets an electric shock, he will connect these two things after a number of combinations. Afterwards, he will become afraid of the sound. A third important function of the amygdala is the consolidation of long-term emotional memories. Prefrontal cortex Just as the amygdala, the prefrontal cortex (PFC) plays a role in reward processing. Neurons in the PFC can detect changes in the reward value of learned stimuli. Furthermore, the PFC is involved in planning, making decisions based on earlier experiences and working towards a goal. The combination of functions of the PFC is described as the `executive function'. Anterior cingulate cortex This part of the brain is generally subdivided into a `cognitive' and a `affective' part. The affective part is suggested to monitor disagreement between the functional state of the organism and any new information that might have affective consequences. Hypothalamus The hypothalamus is the part of the brain that controls many processes in the body, such as body temperature, hunger and thirst. It also handles the release of some hormones. As such, the hypothalamus is involved in processing emotions and sexual arousal. Insular cortex The insular cortex is said to be associated with emotional experience and produces conscious feelings. It combines sensory stimuli to create an emotional context. 2.2.2 Models of emotion Emotion is a phenomenon that is difficult to grasp, as described in section 2.2. In order to classify and represent emotions, some models have been proposed. There are two dominant models. Basic emotions One of the models uses the idea that all emotions can be composed of some basic emotions, just as colours can be composed of primary colours. Plutchik relates eight basic emotions to evolutionary valuable properties, and based thereupon he claims these to be the basic

emotions: anger, fear, sadness, disgust, surprise, curiosity, acceptance, and joy. Ekman has chosen other emotions to be basic, and has found that these emotions and expressions of these emotions are universal. Ekman's list of basic emotions is: anger, fear, sadness, happiness, disgust and surprise. Dimensional view Another model is composed of multiple dimensions, and places every emotion on a multidimensional scale. The first dimension is emotional valence, with positive on the one and negative on the other side. The second axis contains the arousal, from calm to excited. The third dimension is not always used, and when used, it is not always the same. Sometimes dominance is used as the third dimension. Other times it is motor activation (approach or avoid) that is on the third axis.

Figure 2.7: Dimensional model of emotion

This model is used in most studies, because of two reasons Simplicity: it is easy to express an emotion in terms of arousal and valence, whereas it is much more difficult to decompose an emotion into basic emotions. Universality: there is little controversy about the first two dimensions of the model. Valence and arousal are natural terms to use, when speaking about emotion, and they are understood between all cultures.

2.2.3 Measuring emotion using EEG Although we know something about the way the brain handles emotion, we also want to be able to measure this activity. Unfortunately, EEG uses electrodes outside the skull. It is a known problem for EEG measurements that the received electrical signals are not only signals from one spot, but the skull spreads out the brain activity from the whole brain. Because most of the mentioned structures lie deep inside the brain, it is hard to get some information from it using EEG. Fortunately, there have been a lot of people doing research after the measurements of emotion using EEG. One of the most apparent results is the important role of the alpha frequency band. Kostyunina et al. have found that different emotional states show different peak frequencies in the alpha band. More recent work from Shemyakina et al. shows that significant differences in local EEG power and spatial synchronization (EEG power at different spots to behave in the same way) can be observed with different emotions. This effect was largest in the temporal area of the brain. Other researchers have concentrated on one of the two emotional dimensions, and determined how the value in both dimensions could be found from the EEG data. One of the most apparent things about emotion in EEG measurements is the difference between the left and the right hemisphere. It has been suggested that information about the valence dimension can be found in this difference, which is called the hemispherical emotional valence (HEV). Basically, there are two theories about HEV: Right hemisphere hypothesis: this hypothesis states that the right hemisphere is mostly involved in processing emotion. This lateralization should be most apparent with negative emotions. Valence asymmetry hypothesis: This hypothesis poses that the involvement of both hemispheres depends on the valence of the emotion. The right hemisphere is dominant with negative emotions whereas the left hemisphere is more active with positive emotions. Both hypotheses have been supported by a lot of research. However, some recent papers have compared and analyzed most of this research, and have not found empirical evidence for any of the theories. Murphy et al. have performed a meta-analysis on 106 studies on human emotion. They found partial support for the valence asymmetry hypothesis, because greater activity was observed in the left hemisphere for positive emotions, where the activity was symmetrical for negative emotions. Schiffer et al. have collected a lot of literature on this subject, and have found no strong empirical support for any of the theories. They hypothesize that the direction of the hemispherical emotional valence is a personal matter, just like gender or hand-writing. They have found support for this.

Arousal Where there is a lot of information about EEG properties of the valence dimension, less is known about how to measure arousal using EEG signals. Several studies have been done, but the results are less obvious than for the valence dimension. Heller et al. gives an overview of many of these studies. The main conclusion from earlier research shows that the right posterior hemisphere has something to do with arousal. This has also been confirmed by their own research. Aftanas et al. state that not only the right posterior hemisphere is involved with arousal, but also the left anterior hemisphere. They show that emotional states with a high or low arousal value, as compared with neutral signals, show asymmetrical increases in activity in the posterior areas of the right and the anterior areas of the left hemispheres. Krause et al. take another approach, and studied the differences between EEG bands, instead of the brain areas. They showed that the 4-6Hz EEG band elicited a greater synchronization when viewing an aggressive film than when viewing a neutral or sad film. 2.3 Other literature on emotion recognition using physiological signals There have been other researchers that have used physiological signals to detect emotion For example; Choppin has built an emotion recognizer for patients suffering from amyotrophic lateral sclerosis (ALS). These patients lose, at some moment during their disease, their ability to use their muscles, and those patients lose their ability to speak or to communicate on any other way. For that reason Choppin uses only EEG signals to recognize emotion, in order to provide those people with the possibility to express their feelings. He uses neural networks to classify the EEG signals online, and achieved a correct classification rate for new unseen samples of about 64%, when using three emotion classes. Mainly because of the low number of training data, but also because the project was very preliminary, these results could be improved. Musha et al. also use EEG signals to read a persons emotion. They have extracted cross-correlation coefficients between the EEG activities from different locations, to find information about the correlation between the EEG signals from different locations. They have computed an `emotion matrix' to transform these coefficients linearly into a four element vector that corresponds with 4 basic emotions. The numbers in the vector denote how strong that particular emotion is found in the EEG signals. The authors do not mention how well their system performs. Another experiment of Chanel et al. tried to recognize only the arousal dimension of emotion from EEG and other physiological measures. The activity in specific EEG bands on specific locations on the scalp was used as features to describe the EEG data. The authors used different classifiers to classify the EEG signals, the physiological signals and a combination of both. Results showed that the EEG signals performed equally well as the physiological signals, but a combination of both improved the results. Classification rates were around 60% when using two classes and 50% when using three classes. Kim et al have used other physiological measures, such as electrocardiogram (ECG) and skin temperature to build a user-independent system for recognizing emotion. The support vector machine they used to implement the system was able to correctly classify 78% and 62% of the samples, in case of three and four different emotions respectively. Haag et al explore the use of bio-sensors, to find out whether they are suitable for complementing or replacing other modalities in measuring emotion. In particular, the authors use

electromyography (EMG), ECG, skin conductivity, blood volume pulse and respiration rate. Two separate neural networks are trained to predict the arousal and valence values respectively, for one subject. Results that were within 20% of the correct answer were counted as correct. This resulted in 96.6% correct classification for arousal and 89.93% for valence.

2.4 Summary Electroencephalography (EEG) seems to be the most practical way of measuring brain activity, because it is cheap and easy to use. Several researchers have shown that it is possible to measure emotional cues using EEG measurements, which is an important condition to be able to find emotion from EEG activity. In literature, a distinction is made between two dimensions of emotion, the valence dimension ranging from negative to positive and the arousal dimension, ranging from calm to excited. Information about both dimensions has been found to be present in EEG signals, which shows that emotion recognition from EEG signals should be possible. We will also use this distinction in our project.

CHAPTER 3

Tools used

3.1

Emotiv neuro headset

3.1.1 About Emotiv A San Franciscobased neuro engineering company called Emotiv is developing a brain computer interface that they say will be available on the commercial market in 2009. The EPOC neuroheadset uses EEG technology to read electrical patterns in the brain and then sends this information through wireless signals to a computer. According to Emotiv, the headset will be used with new biofeedback games or can be incorporated into popular PC games like Harry Potter, where characters could pick up and move objects with the power of their minds. In addition, the EPOC could eventually be used in multiplayer online games like World of Warcraft or Second Life to control facial expressions of virtual game characters in real time. Emotiv claims the headset can detect and replicate thirty different emotional and facial expressions, including excitement, anger, laughter, and calmness.

Figure 3.1: EMOTIV neuro headset Unlike earlier EEG devices, the EPOC is the first commercially available EEG neuroheadset that does not require gel on the scalp or an elaborate net of electrodes. The EPOC will be bundled with Emortal software, which enables you to use the headset to browse your computer files and applications, and also to connect to other Emotiv users in live chat rooms. However Emotiv is consist of 14 channels bipolar 10-20 electrode system as soon in the given figure.

Figure 3.2: Channel locations 3.1.2 Headset Setup The Headset Setup panel is displayed by default when starting Emotiv Control Panel. The main function of this panel is to display contact quality feedback for the neuroheadsets EEG sensors and provide guidance to the user in fitting the neuroheadset correctly. It is extremely important for the user to achieve the best possible contact quality before proceeding to the other Emotiv Control Panel tabs. Poor contact quality will result in poor Emotiv detection results.

Figure 3.3: Emotiv SDK

The image on the left is a representation of the sensor locations when looking down from above onto the users head. Each circle represents one sensor and its approximate location when wearing the SDK headset. The colour of the sensor circle are presentation of the contact quality. To achieve the best possible contact quality, all of the sensors should show as green. Other sensor colours indicate:

Black Red Orange Yellow Green

No signal Very poor signal Poor signal Fair signal Good signal

3.1.3 Hydrating the Sensors Open the Saline Hydration Sensor Pack with the white felt inserts inside. The inserts will eventually be mounted in the headset arms but must be properly wetted with saline solution first. Begin wetting each of the felt inserts with the supplied saline solution. The felts should be wet to the touch, but not soaking wet!

Figure 3.4: Hydration of sensor

Add a few drops of saline to saturate the large white hydrator pad attached to the top cover of the hydrator, then close the cover and gently shake the hydrator pack. This will maintain the

moisture of the felt pads when they are not in use. Open the pack and check that each of the pads had been wetted. If not fully wetted, then add a drop or two of saline to any pads not sufficiently wet using the dropper bottle. Be careful not to over-wet the pads. If you have connection problems, add more saline to each felt pad.

3.1.4 Sensor Assembly After the wetting process, remove the sensor units with their felt pads from the hydrator pack and insert each one into the black plastic headset arms, turning each one clockwise one quarter turn until you feel a definite "click". The "click" indicates each sensor is correctly installed in a headset arm. If you have difficulty with this step, apply a little more force until you feel the "click" but be careful not to exert excessive force as damage might occur.

Figure 3.5: How to open the sensors

3.1.5 Headset Placement You are now ready to put the EPOC headset on your head. Using both hands, slide the headset down from the top of your head. Place the arms approximately as depicted, being careful to place the sensors with the black rubber insert on the bone just behind each ear lobe. Correct placement of the rubber sensor is critical for correct operation.

Figure 3.6: Way to placement of headset Notice the 2 front sensors should be approximately at the hairline or about the width of 3 fingers above your eyebrows. After headset is in position, press and hold the 2 reference sensors (located just above and behind your ears) for about 5-10 seconds. Good contact of reference sensors is the key for a good signal. Check that the lights corresponding to these 2 reference sensors turn from red to green in the EPOC Control Panel Headset Setup screen.

Figure 3.7: Right way to check the placement

3.2 MATLAB The MATLAB high-performance language for technical computing integrates computation, visualization, and programming in an easy-to-use environment where problems and solutions are expressed in familiar mathematical notation. Typical uses include

Math and computation Algorithm development Data acquisition Modelling, simulation, and prototyping Data analysis, exploration, and visualization Scientific and engineering graphics Application development, including graphical user interface building

MATLAB is an interactive system whose basic data element is an array that does not require dimensioning. It allows you to solve many technical computing problems, especially those with matrix and vector formulations, in a fraction of the time it would take to write a program in a scalar non interactive language such as C or FORTRAN. The name MATLAB stands for matrix laboratory. MATLAB was originally written to provide easy access to matrix software developed by the LINPACK and EISPACK projects. Today, MATLAB engines incorporate the LAPACK and BLAS libraries, embedding the state of the art in software for matrix computation. MATLAB has evolved over a period of years with input from many users. In university environments, it is the standard instructional tool for introductory and advanced courses in mathematics, engineering, and science. In industry, MATLAB is the tool of choice for highproductivity research, development, and analysis. MATLAB features a family of add-on application-specific solutions called toolboxes. Very important to most users of MATLAB, toolboxes allow you to learn and apply specialized technology. Toolboxes are comprehensive collections of MATLAB functions that extend the MATLAB environment to solve particular classes of problems. You can add on toolboxes for signal processing, control systems, neural networks, fuzzy logic, wavelets, simulation, and many other areas. 3.2.1 The MATLAB System The MATLAB system consists of these main parts: Desktop Tools and Development Environment This part of MATLAB is the set of tools and facilities that help you use and become more productive with MATLAB functions and files. Many of these tools are graphical user

interfaces. It includes: the MATLAB desktop and Command Window, an editor and debugger, a code analyzer, and browsers for viewing help, the workspace, and folders. Descriptions of some important features are giving below. The command window The right hand side of the MATLAB desktop contains the Command Window. A user can enter interactive commands at the command prompt (>>) in the command window and they will executed on the spot. The Command History Window The Command History Window displays a list of the commands that a user has entered in the Command Window. The list of previous commands can extend back to previous executions of the program. Command remains in the list until they are deleted. To re-execute any command, simply double click it with the left mouse button. The Edit/Debug Window This is used to create new M-file, or to modify existing ones. You can create a new M file with the File/New/M- File selection from the desktop menu or by pressing CTRL+N by keyboard. The Edit/Debug window is essentially a programming text editor, with the MATLAB languages features highlighted in different colours. Comment in M file are green , variables and number appear in black , character strings appear in red , and language keywords appear in blue. The MATLAB workspace A statement like Z=10; Creates a variable named Z, stores the value 10 in it, and saves it in a part of computer memory known as a workspace. A workspace is the collection of all the variables and array that can be used by MATLAB when a particular command, M file , or functions is executing .

Mathematical Function Library This library is a vast collection of computational algorithms ranging from elementary functions, like sum, sine, cosine, and complex arithmetic, to more sophisticated functions like matrix inverse, matrix Eigen values, Bessel functions, and fast Fourier transforms. The Language The MATLAB language is a high-level matrix/array language with control flow statements, functions, data structures, input/output, and object-oriented programming features. It allows

both "programming in the small" to rapidly create quick programs you do not intend to reuse. You can also do "programming in the large" to create complex application programs intended for reuse. Graphics MATLAB has extensive facilities for displaying vectors and matrices as graphs, as well as annotating and printing these graphs. It includes high-level functions for two-dimensional and three-dimensional data visualization, image processing, animation, and presentation graphics. It also includes low-level functions that allow you to fully customize the appearance of graphics as well as to build complete graphical user interfaces on your MATLAB applications. External Interfaces The external interfaces library allows you to write C and FORTRAN programs that interact with MATLAB. It includes facilities for calling routines from MATLAB (dynamic linking), for calling MATLAB as a computational engine, and for reading and writing MAT-files. To start MATLAB from Windows, double-click the MATLAB icon on your Windows desktop. To start it from a UNIX platform, type MATLAB at the operating system prompt. When MATLAB starts, the MATLAB desktop opens as shown in Figure 1.1. The window in the desktop that concerns us for this chapter is the Command Window, where the special prompt appears. This prompt means that MATLAB is waiting for a command. You can quit MATLAB at any time with one of the following:

Select Exit MATLAB from the desktop File menu. Enter quit or exit at the Command Window prompt.

Do not click on the close box in the top right corner of the MATLAB desktop. This does not allow MATLAB to terminate properly and, on rare occasions, may cause 3.2.2 Advantage of MATLAB MATLAB has many advantages compared with conventional computer languages, for technical problem solving, among them are the following 1- Ease of use MATLAB is an interpreted language, like many versions, it is very easy to use . The program can be used as a scratch pad to evaluate expressions typed at the command line, or it can be used to execute large prewritten programs. Programs may be easily written and modified with a built in integrated development environment and debugged with the MATLAB debugger.

2- Platform independence MATLAB is supported on many different computer systems, providing a large measure of platform independence. At the time of this writing, the language is supported on all versions of windows and many different versions of UNIX.

Figure 3.8: A typical MATLAB window

3. Predefined Functions MATLAB comes complete with an extensive library of pre- defined functions that provide tested and pre-packaged solutions to many basic technical tasks. 4. Graphical user interface MATLAB includes tools that allow a programmer to interactively construct a graphical user interface (GUI) for his program .With this capability, the programmer can design sophisticated data analysis programs that can be operated by relatively inexperienced users.

3.3 Emotional stimulation For emotional stimulation we will use pictures from the IAPS. However, some of the images were found not to evoke the right emotions, when comparing the IAPS annotations with the self-assessments. These images were excluded from this experiment, resulting in 78 image series. The full list of images used in this experiment is found in appendix C.

Figure 3.9: An image similar to IAPS images A digitized version of the Self Assessment Manikin (SAM) was used to evaluate the emotion that was experienced by the participant during the time he saw the pictures. The version contained a 5-point scale for both the valence and the arousal

Figure 3.10: Self assessment manikin used in experiment

The International Affective Picture System (IAPS) is being developed to provide ratings of affect for a large set of emotionally-evocative, internationally-accessible, colour photographs that include contents across a wide range of semantic categories. The IAPS (pronounced EYE-APS), along with the International Affective Digitized Sound system (IADS), the Affective Norms for English Words (ANEW), as well as other collections of affective stimuli, are being developed and distributed by the NIMH Center for Emotion and Attention (CSEA) at the University of Florida in order to provide standardized materials that are available to researchers in the study of emotion and attention. The existence of these collections of normatively rated affective stimuli should: 1) allow better experimental control in the selection of emotional stimuli, 2) facilitate the comparison of results across different studies conducted in the same or different laboratory, and 3) encourage and allow exact replications within and across research labs who are assessing basic and applied problems in psychological science.

CHAPTER 4

METHODOLOGY

4.1 DATA ACQUISITION 4.1.1 Experimental protocol The participant is given information about the experiment beforehand. He has to watch the images that are shown, and remember his emotional state when he sees the images. He also has to minimize his muscle and eye movements. He is asked to take a seat in the chair, and watch the screen. When the person is seated and wears the Emotiv, one test session is performed by showing one series of images and the SAM. This way, the participant will become familiar with the task he has to perform. The real experiment consists of 30 images, with a pause of about 12 seconds in between. First ally 3minute baseline is recorded with eye open then 6 seconds for one image and lastly 12 second for self assessment. The orders of images are Happy, Calm, Fear, Disgust, Surprise and this process continue till 30 images. So total 12 minute recording are taken for each subject and total 10 subjects recording are done.

BASELINE (EYE OPEN) 3MINUTES

SINGLE IMAGE 6 SECONDS Table 4.1: Protocol description

BLACK SCREEN WITH CROSS 12 SECONDS

4.1.2Participants 10 male participants were found to voluntarily participate in the experiments

Figure 4.1: LAB Setup

All participants are trainee at INMAS, which have no prior knowledge of this type of experiment .The participants were between 19 and 29 years old. Every participant was given information about the experiment, and was asked to fill in an informed consent, stating that they participated voluntarily and that they had enough knowledge about the experiment. During the recording following window is popup in which we can see 14 channel EEG data recording process in progress.

Figure 4.2: 14 channels EEG recording 4.2 PREPROCESSING Pre-processing is a step to process raw EEG signals in such a way that they are ready to be used. As depicted in figure, the measured EEG signal is a combination of brain activity, reference activity and noise. The goal of this pre processing step is to reconstruct the original brain activity. All steps can be executed using different methods and algorithms. These steps will be explained in this section. 4.2.1 Referencing EEG signals are a recording of the voltage at different electrodes. Ideally, this measurement should only represent the electrical activity on that spot. However, since voltage is a relative measure, the measurement is compared to the voltage at a reference site. Unfortunately, this results in measurements that reflect the local activity, but also the activity at the reference site. Because of this, the reference should be

Figure 4.3: EEG signals as a combination of brain activity, reference activity and noise Chosen such that the activity at the reference site is almost zero. The nose, mastoids and earlobes are typically used some researchers have stated that the differences between results of different studies are partly due to the differences in referencing. This underlines the importance of the subject. Common reference A widely used method of referencing is the common reference technique. This method uses one common reference for all electrodes, preferably a reference at a large distance from all electrodes. The activity at the reference site influences all measurements equally, and differences between electrode measurements still contain all information needed. Average reference Another method is the average reference. The average reference subtracts the average of the activity at all electrodes from the measurements. This method is based on the principle that the activity at the whole head at every moment sums up to zero. Therefore, the average of all activity represents an estimate of the activity at the reference site. Subtracting this average produces in principle a de referenced solution. However, the relatively low density of the electrodes and the fact that the lower part of the head is not taken into account, bring some practical problems along. Current source density A technique that is frequently recommended for asymmetry research it the current source density (CSD) approach. The current source density is "the rate of change of current owing into and through the scalp". This quantity can be derived from EEG data, and it may be interpreted as the potential difference between an electrode and a weighted average of their surrounding electrodes. The CSD can be estimated by computing the laplacian. This comes down to computing the sum of the differences between an electrode and its neighbours. A problem with this estimation is that it is actually only valid when the electrodes are in a 2d plane and equally distant. Despite of this problem, the CSD is used in many studies.

Conclusions Even though there is a lot of literature available about referencing, it is still not very clear which method to use. All techniques have some problems in practice, and thus one can only choose the least bad one. Preliminary research from Hagemann et al. points to the CSD method as the most promising method, when researching hemispherical asymmetry. We will use this method in our project. 4.2.2 Noise and artifacts The brain produces electrical activity in the order of micro volts. Because these signals are very weak, it usually contains a lot of noise. Sources of noise are static electricity or electromagnetic fields produced by surrounding devices. In addition to this external noise, the EEG signal is most of the times heavily influenced by artifacts that originate from body movement, as depicted in figure 3.2. For example, eye blinks or other eye movements produce large spikes in the EEG signal. Other muscle movements also leave their mark in the brain signal. In many studies the participants are asked not to move or to blink as few times as possible. However, in many practical situations this is not feasible.

Figure 4.4: EEG signal contaminated with artifacts. The red boxes show the artifacts 4.2.3 Wavelet one dimensional De noising wden is also known as one-dimensional de-noising function. wden performs an automatic denoising process of a one-dimensional signal using wavelet. [XD, CXD, LXD]= wden(X, TPTR, SORH, SCAL, N, wname)

returns a de-noised version XD of input signal X obtained by thresholding the wavelet coefficients. Additional output arguments [CXD, LXD] are the wavelet decomposition structure of the de-noised signal XD. TPTR string contains the threshold selection rule: 1) rigrsure uses the principle of Steins Unbiased Risk. 2) heursure is an heuristic variant of the first option. 3) sqtwolog for universal threshold (2log(.))^1/2 4) minimaxi for minimax thresholding. SORH (s or h) is for soft or hard thresholding rescaling: 1) one for no rescaling 2) sln for rescaling using a single estimation of level noise based on first level coefficients 3) mln for rescaling done using level-dependent estimation of level noise

Figure 4.5: An EEG signal after 1-D de noising

4.2.4 Filtering Many noises that are present in the EEG signals can be removed using simple filters. The relevant information in EEG, at least for emotion recognition, is found in the frequencies below 30Hz. Therefore, all noise with higher frequencies can be removed using a low pass filter. For example, noise from the electrical net has a fixed frequency of 50Hz. Bandpass filters can also divide the EEG signals into frequency bands which can be analyzed separately. Low pass and band pass filters are implemented in MATLAB using FIR filters. Filter design in MATLAB is done by fdesign command of MATLAB. Filter design through user-defined specifications is the core of the fdesign approach. This specification-centric approach places less emphasis on the choice of specific filter algorithms, and more emphasis on performance during the design a good working filter. For example, you can take a given set of design parameters for the filter, such as a stop band frequency, a pass band frequency, and a stop band attenuation, and using these parameters design a specification object for the filter. You can then implement the filter using this specification object. Using this approach, it is also possible to compare different algorithms as applied to a set of specifications. There are two distinct objects involved in filter design:

Specification Object Captures the required design parameters of a filter Implementation Object Describes the designed filter; includes the array of coefficients and the filter structure

The distinction between these two objects is at the core of the filter design methodology.

Figure 4.6: Magnitude response of a typical band pass filter You can run the code from the Help browser (select the code, right-click the selection, and choose Evaluate Selection from the context menu), or you can enter the code on the MATLAB command line. Before you begin this example, start MATLAB and verify that you

have installed the Signal Processing Toolbox. If you wish to access the full functionality of fdesign and filterbuilder, you should additionally obtain the Filter Design Toolbox software. You can verify the presence of these products by typing ver at the command prompt Wavelet filters [Lo_D,Hi_D,Lo_R,Hi_R] = wfilters(wname) computes 4 filters related to the orthogonal or biorthogonal wavelet named in the string wname The 4 output filters are: 1) Lo_D, the decomposition low pass filter 2) Hi_D, the decomposition high pass filter 3) Lo_R, the reconstruction low pass filter 4) Hi_R, the decomposition high pass filter 4.2.5 Artifacts Artifacts are somewhat more difficult to remove, because they are not present all the time, and not always in all electrodes. Another disadvantage of artifacts over other noise is the relative high voltage as compared to the normal EEG signals. However, since artifacts contaminate the signal very much, they are a huge problem when using EEG signals. Fortunately, for the same reason many solutions for the problem have been suggested, ranging from very simple to mathematically complex. Filtering A very simple method of removing artifacts is simply using a high pass filter to remove the frequencies below 1 or 2Hz. This method assumes there is not much brain activity with very low frequencies, and that artifacts occur at a lower frequency. A great advantage of this method is that its implementation is very simple and it can be combined with the low pass filtering to remove noise with high frequencies. Unfortunately, the method is not very precise, and might remove some useful information. Artifact rejection One method of removing artifacts is the rejection of the data sample. Although this is a valid method of removing artifacts, the problem is that there are often not much samples available. If some of the samples are removed, there are even less samples left, which might decrease the quality of the system. Another problem of rejecting data in online systems is the possibility of losing some important part of the data, which might not return again. Artifact subtraction Another method involves the subtraction of electrooculographic (EOG) signals from the EEG data .EOG measures the eye movement. For each EEG channel, a scale factor is estimated for how much it is involved by eye movements. The EOG signal is scaled with this factor, and subtracted from the EEG signal. The main disadvantage of this method is the fact that the EOG signal contains traces of brain activity, especially from the frontal lobe. This activity is also subtracted from the EEG signals, and some useful information is removed.

Adaptive filtering

The use of adaptive filtering in another method for the online removal of EOG artifacts. This algorithm estimates the clean EEG signals by subtracting the filtered EOG signals from the measured EEG signal. The filter is adjusted to the signal automatically, and therefore does not need any calibration or learning steps. He et al. have investigated the use of adaptive filtering for EOG artifact removal. They report good artifact removal and a good and fast convergence of the algorithm, which makes it suitable for online filtering. Blind source separation Another more difficult method is using Blind Source Separation (BSS). BSS decomposes signals from multiple sources, in the case of EEG the electrodes, into another set of signals, such that the signals' independency is maximized. This method has the advantage that is does not need separate EOG measurements. BSS relies on the assumption that these independent sources indeed exist. For EEG signals, this assumption holds. As the EEG signals on an electrode are said to be a mix of underlying signals of brain processes, the signals can be decomposed. An example of BSS is shown in figure 3.3, where all different samples can be coarsely separated into two groups, corresponding to the blue and the red line.

FIGURE 4.7: Example of BSS two components are found within the data A classic example of BSS that might clarify this method is the cocktail party problem. A number of people are talking in a room, and this is recorded with some microphones. These microphones record all voices together, and the goal of BSS is to identify the underlying sources of the sound, in this case the different voices. Independent Component Analysis A well-known and widely used method of BSS is Independent Component Analysis (ICA). ICA assumes that the components are statistically independent at each time point, but does not consider the relationships across time. This method is very powerful and some fast implementations exist. However, when separating EEG signals, the user has to manually distinguish the real brain signals from the artifacts. Mainly ICA is done in EEGLab.

Second Order Blind Inference To use a particular method for removing ocular artifacts in an online system, it has to be able to automatically remove the artifacts, without manual intervention. This method makes other assumptions on the components. Automatic removal without EOG measurements Unfortunately we do not have the equipment to measure EOG signals separately. This is needed for most before mentioned methods. BSS is capable of removing artifacts without the separate EOG measurement. However, that method requires manual intervention, and the automatic method described earlier in this paragraph does need the EOG measurement again. One attempt was made using an extended Kalman filter is used to detect dynamic changes in the signal. Using that information they were capable of detecting artifacts. The main disadvantage of this method is that is seems to work well on muscle artifacts, but not so well on ocular artifacts. Furthermore, it is a computationally complex method. Another solution proposed by Delorme et al. He is using higher-order statistics on ICA components, in order to select some pieces of the EEG signal for rejection. They compute the entropy and the kurtosis for the ICA components. However, the authors still check the selected pieces manually, in order to avoid mistakes. Nicolaou et al. went in the same direction. They used Temporal Decorrelation Source Separation a temporal extension to the standard ICA algorithm, to separate the EEG from artifacts. Using a support vector machine the different components could be divided in artifact-related components and components containing useful information. 4.3 Feature extraction Feature extraction is the process of extracting useful information from the signal. Features are characteristics of a signal that are able to distinguish between different emotions. For example, when trying to assess the health of a person, you can check his weight, heartbeat rate, blood pressure etc. Those measures contain a lot of information about the health of a person. However, the length of that same person (most of the times) does not have anything to do with his health. All features together should contain all information that is needed for the specified task. Since there are many features that could be extracted, the difficulty is which features to extract from the signal. The features we will use are based on earlier research. Several studies on recognizing emotion have already been performed. The studies all use different features and claim good results using those features. However I have used peak frequency and energy to classify for emotions.
.

4.3.1 Peak frequency in alpha band Some researchers have found that the peak frequency (i.e. the frequency of the maximal amplitude peak of the power spectrum) in the alpha band can differentiate between emotions. This peak frequency has a tendency to increase with joy and anger and to decrease for sorrow and fear, as compared to a baseline measurement. This measure will also be used in our project. 4.3.2 Energy of alpha band using Wavelet A wavelet is a small oscillatory wave which has its energy concentrated in time. It has an ability to allow simultaneous time and frequency analysis and it is a suitable tool to analyze transient, non-stationary or time varying signals. The non-stationary nature of EEG signals is to expand them onto basis functions created by expanding, contracting and shifting a single

prototype function (a,b, the mother wavelet), specifically selected for the signal under consideration The mother wavelet function a,b (t) is give as

where a, b R, a>0, and R is the wavelet space. Parameter 'a' is the scaling factor and b is the shifting factor. The only limitation for choosing a prototype function as mother wavelet is to satisfy the admissibility condition,

FEATURES Energy Recursing energy efficiency(REE) Logarithmic Recursing energy efficiency(LREE)

DESCRIPTION Energy of alpha band calculated using db4 wavelet It is alpha band energy divided by total energy It is log base 10 of REE.

Table 4.2: Statistical features used for emotion recognition and their description Frequency Range 0-4 4-8 8-16 16-32 32-64 Decomposition Level A5 D5 D4 D3 D2 Frequency Bands DELTA THETA ALPHA BETA GAMMA Frequency Hz 4 4 8 16 32

Table 4.3: Decomposition of EEG signals into different frequency bands with a sampling frequency of 128 Hz

Decomposition using wavedec function Wavedec performs multilevel one dimensional wavelet analysis by using a specific wavelet decomposition filters(Lo_D and Hi_D). [C,L] = wavedec (X,N,wname) returns the wavelet decomposition of the signal X at level N, using wname. N must be strictly a positive integer. The output decomposition structure contains the wavelet decomposition vector C and the book keeping vector L. The structure is organized as in this level 3 decomposition example

Figure 4.8: Level 3 decomposition 4.4 Classification After extracting the desired features, we still have to find the emotion. This process will be done by a classifier. A classifier is a system that divides some data into different classes, and is able to learn the relationship between the features and the emotion that belongs to that part of the EEG signal. Several methods for classification have been proposed. We will explain some of the most well-known algorithms. 4.4.1 Artificial Neural Networks The human brain consists of billions of neurons. These neurons are connected to thousands of other neurons, and together form a neural network. Through these connections, the neurons are able to communicate with each other, and as a whole the network is able to perform highly intelligent actions, such as memorizing things or controlling parts of the human body. (Artificial) neural networks (ANN or NN) originate from the desire to mimic the human brain in a computer. ANN s can be seen as a simplified model of the human brain and also consists of several neurons, where some neurons are connected. With all neurons together, an ANN tries to learn a relationship between the input and the output, without having knowledge about the underlying model.

Figure 4.9: Multilayer perceptron neural network

Figure 4.10: The structure of a neural network The neurons are also referred to as processing elements (PE), as they process the inputs given to it, combine all values and produce one output value. ANNS are typically organized in several layers. One input layer and one output layer are always present, but one or more hidden layers can be added. The neurons from one layer are connected to the neurons in the next layer, giving its output as input to the next layer neurons. An example of a neural network can be seen in figure 3.5. The input to the input layer is the original data to be classified, in this case the feature values. The outputs from the output layer are the values on the arousal and valence scale. The neuron in an ANN performs a function on the linear summation of the inputs. The weights of the summation are adapted during the training phase, and reflect the importance of the input to this neuron. Typically, the process in the neuron will look like y(x) = f(s(x)) where s(x) is the linear combination of inputs x with weights w and bias b:

The neural networks do not need to know the underlying model of the data. Instead, an ANN is a general structure that is capable of learning the relation between input and output itself. The most widely used training scheme is the back-propagation method. This method uses the difference between the output of the ANN and the expected output to change the weights w for all neurons in output layer. This process is repeated for the other layers in the network.

The total training consists of the repetition of this training step, where the weights are improved every step. The back-propagation training is performed in the following way:

Where w (n) is the weight vector for a certain neuron at training step n, d is the desired output of that neuron, y is the actual output of the neuron, x is the input vector and is the learning vector. In every training step the difference between the desired output d and the actual output y will be smaller. Therefore, the training will converge to the optimal weights. In an ANN there is always one input layer, with as many neurons as there are features. The number of neurons in the output layer is the same as the number of desired outputs or classes. The number of hidden layers can be chosen freely. Using one or more hidden layers gives the ANN the capability of learning more complex relations between the input and output. However, it also increases the number of samples needed to train the system. Within the hidden layers, the number of neurons can also be chosen freely. More neurons in a layer can result in a better overall result, but as with the number of layers, the number of samples needed for training also increases with a higher number of neurons in a layer.

4.4.2 Fuzzy C Means Fuzzy The concept of Fuzzy Logic (FL) was conceived by Lotfi Zadeh, a professor at the University of California at Berkley, and presented not as a control methodology, but as a way of processing data by allowing partial set membership rather than crisp set membership or non membership. This approach to set theory was not applied to control systems until the 70's due to insufficient small-computer capability prior to that time. Professor Zadeh reasoned that people do not require precise, numerical information input, and yet they are capable of highly adaptive control. If feedback controllers could be programmed to accept noisy, imprecise input, they would be much more effective and perhaps easier to implement. Unfortunately, U.S. manufacturers have not been so quick to embrace this technology while the Europeans and Japanese have been aggressively building real products around it. In this context, FL is a problem-solving control system methodology that lends itself to implementation in systems ranging from simple, small, embedded micro-controllers to large, networked, multi-channel PC or workstation-based data acquisition and control systems. It can be implemented in hardware, software, or a combination of both. FL provides a simple way to arrive at a definite conclusion based upon vague, ambiguous, imprecise, noisy, or missing input information. FL's approach to control problems mimics how a person would make decisions, only much faster. FL incorporates a simple, rule-based IF X AND Y THEN Z approach to a solving control problem rather than attempting to model a system mathematically .The FL model empirically-based, relying on an operator's experience rather than their technical understanding of the system. For example, rather than dealing with temperature control in terms such as "SP =500F", "T <1000F", or "210C <TEMP <220C", terms like "IF (process is too cool) AND (process is getting colder) THEN (add heat to the process)" or "IF (process is too hot) AND (process is heating rapidly) THEN (cool the process quickly)" are used. These terms are imprecise and yet very descriptive of what must actually happen. Consider what you do in the shower if the temperature is too cold: you will make the water comfortable very

quickly with little trouble. FL is capable of mimicking this type of behaviour but at very high rate. FL requires some numerical parameters in order to operate such as what is consider significant error and significant rate-of-change-of-error, but exact values of these numbers are usually not critical unless very responsive performance is required in which case empirtuning would determine them. For example, a simple temperature control system could use a single temperature feedback sensor whose data is subtracted from the command signal to compute "error" and then time-differentiated to yield the error slope or rate-of-change-of-error, hereafter called "error-dot". Error might have units of degs F and a small error considered to be 2F while a large error is 5F. The "error-dot" might then have units of degs/min with a small error-dot being 5F/min and a large one being 15F/min. These values don't have to be symmetrical and can be "tweaked" once the system is operating in order to optimize performance. Generally, FL is so forgiving that the system will probably work the first time without any tweaking. FUZZY LOGIC (FL) was conceived as a better method for sorting and handling data but has proven to be a excellent choice for many control system applications since it mimics human control logic. It can be built into anything from small, hand-held products to large computerized process control systems. It uses an imprecise but very descriptive language to deal with input data more like a human operator. It is very robust and forgiving of operator and data input and often works when first implemented with little or no tuning. Fuzzy C Clustering The Algorithm Fuzzy c-means (FCM) is a method of clustering which allows one piece of data to belong to two or more clusters. This method (developed by Dunn in 1973 and improved by Bezdek in 1981) is frequently used in pattern recognition. It is based on minimization of the following objective function:

, Where m is any real number greater than 1, uij is the degree of membership of xi in the cluster j, xi is the ith of d-dimensional measured data, cj is the d-dimension center of the cluster, and ||*|| is any norm expressing the similarity between any measured data and the center. Fuzzy partitioning is carried out through an iterative optimization of the objective function shown above, with the update of membership uij and the cluster centers cj by:

, This iteration will stop when , where is a termination criterion between 0 and 1, whereas k are the iteration steps. This procedure converges to a local

minimum or a saddle The algorithm is composed of the following steps:

point

of Jm.

1. Initialize U=[uij] matrix, U(0) 2. At k-step: calculate the centers vectors C(k)=[cj] with U(k)

3. Update U(k) , U(k+1)

4. If || U(k+1) - U(k)||<

then STOP; otherwise return to step 2.

Remarks As already told, data are bound to each cluster by means of a Membership Function, which represents the fuzzy behaviour of this algorithm. To do that, we simply have to build an appropriate matrix named U whose factors are numbers between 0 and 1, and represent the degree of membership between data and centers of clusters. For a better understanding, we may consider this simple mono-dimensional example. Given a certain data set, suppose to represent it as distributed on an axis. The figure below shows this:

Figure 4.11: Example of Cluster Data Looking at the picture, we may identify two clusters in proximity of the two data

Figure 4.12: Data cluster and Membership function

concentrations. We will refer to them using A and B. In the first approach shown in above: tutorial - the k-means algorithm - we associated each datum to a specific centroid; therefore, this membership function looked like this In the FCM approach, instead, the same given datum does not belong exclusively to a well defined cluster, but it can be placed in a middle way. In this case, the membership function follows a smoother line to indicate that every datum may belong to several clusters with different values of the membership coefficient.

Figure 4.13: Progress of membership function In the figure above, the datum shown as a red marked spot belongs more to the B cluster rather than the A cluster. The value 0.2 of m indicates the degree of membership to A for such datum. Now, instead of using a graphical representation, we introduce a matrix U whose factors are the ones taken from the membership functions:

The number of rows and columns depends on how many data and clusters we are considering. More exactly we have C = 2 columns (C = 2 clusters) and N rows, where C is the total number of clusters and N is the total number of data. The generic element is so indicated: uij. In the examples above we have considered the k-means (a) and FCM (b) cases. We can notice that in the first case (a) the coefficients are always unitary. It is so to indicate the fact that each datum can belong only to one cluster. Other properties are shown below:

Actually FCM is a data clustering technique wherein each data point belongs to a cluster to some degree that is specified by a membership grade. This technique was originally introduced by Jim Bezdek in 1981 as an improvement on earlier clustering methods. It provides a method that shows how to group data points that populate some multidimensional space into a specific number of different clusters. Fuzzy Logic Toolbox command line function fcm starts with an initial guess for the cluster centers, which are intended to mark the mean location of each cluster. The initial guess for these cluster centers is most likely incorrect. Additionally, fcm assigns every data point a membership grade for each cluster. By iteratively updating the cluster centers and the membership grades for each data point, fcm iteratively moves the cluster centers to the right location within a data set. This iteration is based on minimizing an objective function that represents the distance from any given data point to a cluster center weighted by that data point's membership grade.

Figure 4.14: Fuzzy c clustering in MATLAB

The command line function fcm outputs a list of cluster centers and several membership grades for each data point. You can use the information returned by fcm to help you build a fuzzy inference system by creating membership functions to represent the fuzzy qualities of each cluster

4.4.3 KNN Classifier

The intuition underlying Nearest Neighbour Classification is quite straight forward; examples are classified based on the class of their nearest neighbours. It is often useful to take more than one neighbour into account so the technique is more commonly referred to as k-Nearest Neighbour (k-NN) Classification where k nearest neighbours are used in determining the class. Since the training examples are needed at run-time, i.e. they need to be in memory at run-time; it is sometimes also called Memory-Based Classification. Because induction is delayed to run time, it is considered a Lazy Learning technique. Because classification is based directly on the training examples it is also called Example-Based Classification or Case-Based Classification. In this example the decision for q1 is straightforward all three of its nearest neighbours are of class O so it is classified as an O. The situation for q2 is a bit more complicated at it has two neighbours of class X and one of class O. This can be resolved by simple majority voting or by distance weighted voting. So kNN classification has two stages; the first is the determination of the nearest neighbours and the second is the determination of the class using those neighbours.

Figure 4.15: KNN classifier in MATLAB

CHAPTER 5

RESULT AND DISCUSSION

5.1 Result In this project, we briefly report the earlier studies on emotion recognition using EEG. We are aware of the fact that, emotion detection using EEG is a difficult issue compared to other physiological signals. However, there exists a strong motivation in analyzing EEG signals for emotion classification since emotion and brain activity are highly interrelated. The major issues to be focused on this field are: methodology imposed on placing the electrodes on scalp, time duration and precision in eliciting emotion using different experimental protocol and the type of emotion elicitation methods used. Besides data acquisition, it is also important to consider proficient feature extraction methods for extracting the subtle information from the EEG signal at micro-second time resolution. We have made an effort in studying the feasibility of using Alpha band features under wavelet analysis for improving emotion classification. Classification of emotion, not only depends on the choice of the classifier, but also on the choice of the proper EEG features. In this study, the db4 wavelet function is used for decomposing the EEG signals into a set of five frequency sub bands (Table 3). We considered the statistical features on alpha for classifying the emotions. The values of frequency over different frequency sub bands are summarized in Table 2. From the self assessment section of experiments, it is found that four emotions (disgust, fear, happy and surprise) are strongly experienced by subjects among five emotions. Hence, these four emotions are considered throughout our analysis. Three statistical features (energy, REE and LREE) for the EEG data of ten subjects over four emotions are considered for classifying the emotions using FCM. We have used three clustering validity measure for determining the significance of the statistical features. The main concern on classification ability depends on minimizing the validity measure in such a way to have distinct clusters with maximum parting distance. Among the three statistical features, the REE has a minimum value of the performance measures and it is proficiently clustered the four emotions into a distinct clusters. Figure 5.1 Figure 5.2 present the FCM clustering plots for the two statistical features. These clustering plots show that the EEG data contains enough information to recognize emotion; however a lot of diversity among different people and circumstances exists. In Figure 5.1, the energy over disgust and surprise emotion has lot of overlapping characteristics. The centroids of above emotions are closer to each other than other emotions. In Figure 5.2, each emotion will have larger separation distance between the centroids. All four emotions have individual cluster with minimum overlapping characteristics. From the above clustering plots, we found that, REE feature can efficiently cluster all four emotions with minimal overlapping. In addition, we used three validity measures on measuring the clustering analysis of four emotions with We varied the number of cluster from 1 to 5 and we

achieved our performance goal at 5th cluster, which is From this study, we found that REE is a reasonable feature and hence this is considered as the starting point to proceed. Initial results presented here, have shown motivating results for emotion detection under visual stimulus environment, warranting the need to pursue this idea further. In this current research scenario, we are progressing to update the dataset with increased subjects and to test all the methods with increased number of data sets.

Figure: 5.1: FCM clustering plot for energy feature

Figure: 5.2 FCM clustering of emotions using REE

5.2 CONCLUSIONS In past decade, many researchers have used the physiological signals based emotion assessment. However, very few researchers have considered the EEG as primary indicator for detecting the emotions. The accuracy of classification depends not only on the feature selected for classification but also depends on: (a) number of electrodes used for acquiring the EEG signal (b) number of subjects participated in the experiment (c) nature of stimuli used for evoking the emotions (d) type of classifier used for classification. Many of the works are employ visual stimuli for evoking the emotions with IAPS. The results presented in Section 5.1 prove that, the emotion recognition using EEG is feasible by combining wavelet analysis and FCM. We have conducted this experiment with 10 subjects. The opening results presented in the project show the encouraging outcome for emotion detection under visual stimulus environment, justifying the need to pursue this idea further. In addition, we are also planning to update the dataset with increased number of subjects and scheduling to test all the methodologies with increased number of data sets.

APPENDIX A LIST OF FIGURES FIGURE 1.1 Different emotions FIGURE1.2 The human brain FIGURE2.1 Example of PET scanning system FIGURE2.2 Example of fMRI scanning system FIGURE2.3 The international 10-20 system FIGURE2.4 Nomenclature of electrodes FIGURE2.5 Bipolar and unipolar recording technique FIGURE2.6 The emotional brain FIGURE2.7 Dimensional model of emotion FIGURE3.1 Emotiv Neuro headset FIGURE3.2 Channel locations of Emotiv FIGURE3.3 Emotiv SDK FIGURE3.4 Hydration of sensors FIGURE3.5 How to open the sensor FIGURE3.6 Way to placement of headset FIGURE3.7 Right way to check the placement FIGURE3.8 A typical MATLAB window FIGURE3.9 An image similar to IAPS images FIGURE3.10 Self assessment manikin used in experiment FIGURE4.1 Lab setup FIGURE4.2 14 channels EEG recording FIGURE4.3 EEG signal as a combination of brain activity FIGURE4.4 EEG signal contaminated with artifacts FIGURE4.5 An EEG signal after 1-D de noising 1 2 7 8 10 11 11 12 14 18 19 19 20 21 22 22 26 27 27 29 30 31 32 33

FIGURE4.6 Magnitude response of typical band pass filter FIGURE4.7 Example of BSS two components are found within the data FIGURE4.8 Level 3 decomposition FIGURE4.9 Multilayer perceptron neural network FIGURE4.10 The structure of Neural network FIGURE4.11 Example of cluster data FIGURE4.12 data cluster and FIGURE4.13 Membership function in progress FIGURE4.14 FCM in MATLAB FIGURE4.15 KNN in MATLAB FIGURE5.1 FCM clustering plot for energy FIGURE5.2 FCM clustering plot for REE

34 36 39 40 40 43 44 44 45 46 48 48

APPENDIX B LIST OF TABLES 4.1 Protocol description 4.2 Statistical features used for emotion recognition and their description 4.3: Decomposition of EEG signals into different frequency bands with a sampling frequency of 128 Hz 29 39

39

APPENDIX D BIBILOGRAPHY

1. M. Wessel. Pioneering research into brain computer interfaces. Master's thesis, University of Technology, 2006

Delft

2.L.I. Aftanas, N.V. Reva, A.A Varlamov, S.V. Pavlov, and V.P. Makhnev. Analysis of evoked EEG synchronization and desynchronization in conditions of emotional activation in humans: Temporal and topographic characteristics. Neuroscience and Behavioural Physiology, 34(8):859{867, 2004. 3.M. M. Bradley and P. J. Lang. Measuring emotion: The self-assessment manikin (SAM) and the semantic differential. Journal of Experimental Psychiatry & Behaviour Therapy, 25(1):49{59, 1994. 4.G. Bush, P. Luu, and M.I. Posner. Cognitive and emotional inuencesinanteriorcingulate cortex. Trends in cognitive sciences, 4:215{222, 2000. A. Delorme and S. Makeig 5.. EEGLab: An open source toolbox for analysis of single-trial EEG dynamicsincluding independent component analysis. Journal of Neuroscience Methods, 134:9{21, 2003. 6. A. Delorme, S. Makeig, and T. Sejnowski. Automatic artifact rejection for EEG data using high-order statistics and independent component analysis. In Proceedings of the Third International ICA Conference, 2001. 7.Emotiv. Emotiv website. http://www.emotiv.com/, 2008 8.D. Hagemann, E. Naumann, and J.F. Thayer. The quest for the EEG reference revisited: A glance from brain asymmetry research. Psychophysiology, 38:847857, 2001. 9.K.H. Kim, S.W. Bang, and S.R. Kim. Emotion recognition system using shortterm monitoring of physiological signals. Medical and Biological Engineering and Computing, 42:419{427, 2004. 10. D.L.Weber. Scalp current density and source current modelling. http://dnl.ucsf.edu/users/dweber/dweber_docs/eeg_scd.html, 2001. Visited on September, 19,2007. 11. J. Villegas. Definition of emotion. student/97_fall/theory/emotion/ea/definition.html, 19 http://www.ciadvertising.org/studies/

12.Emotiv. EPOC. s.l. : http://www.emotiv.com/index.php.

13. Bos, Danny Oude. EEG-based Emotion Recognition - The Influence of Visual and AUditory Stimuli 14. Karim Ansari-Asl, Guillaume Chanel and Thierry Pun. A Channel Selection Method For EEG Classification in Emotion Assessment Based on Synchronization Likelihood. 15. Kulikov, M. B. Kostyunina and M. A. Frequency Characteristics of EEG Spectra in the Emotions. s.l. : Neuroscience and Behaviroal Physiolohy, 1996, Vol. 26. 16. M Murugappan, R Nagarajan, and Sazali Yaacob. Appraising Human Emotions using Time Frequency Analysis based EEG Alpha Band Features. Malaysia : 2009 Conference on Innovative Technologies in Intelligent Systems and Industrial Applications, 2009. 17. Thomas Baumgartner, Michaela Esslen, Lutz Jncke. From emotion perception to emotion experience: Emotions evoked by pictures and classical music. 18. Scherer, Klaus R. What are emotions? And how can they be measured? 44(4), 693727, 2005, Social Science Information, Vol. Trends and developments: research on emotion 19. So-Youn Park, Ju-Jang Lee. EEG Feature Extraction and Classification Using Data Dimension Reduction. Korea : IEEE International, 2008. 20.Chu, Kelvin. complex-morlet.html. http://kelvinchufei.blogspot.com/2008/01/wavelet-scalogram-using-

21.University, Departement of Statistic - Stanford. WaveLab850 http://wwwstat.stanford.edu/~wavelab/. 22. http://www.mathworks.com 23. http://www.matlabgeeks.com 24.